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FOREWORD 


The  overall  goal  of  this  research  effort  was  to  develop  and 
demonstrate  a method  for  reclassifying  General  Detail  Navy  Per- 
sonnel, i.  e. , those  who  did  not  qualify  for  assignment  to  an  "A" 
school  on  the  basis  of  aptitude  scores  derived  from  a conventional 
classification  battery.  Recommendations  for  reclassification  were 
based  upon  an  assessment  center  approach  in  which  job  learning 
tasks  and  job  sample  tests  were  utilized  in  determining  an  individu- 
al's ability  to  perform  successfully  in  six  Navy  ratings.  Results 
from  this  work  will  be  employed  in  future  research  and  development 
to  design,  test,  and  evaluate  a classification  system  for  General  De- 
tail personnel. 


The  monitor  for  this  contract  was  Mr.  Paul  Foley.  Mr.  Ted 
Yellen  also  contributed  by  providing  practical  assistance  and  advice 
in  the  establishment  and  conduct  of  the  assessment  center.  Appreci- 
ation is  also  extended  to  LCDR  Gary  Johnson  and  to  CPO  John  Gavel 
of  the  Apprenticeship  Training  Department  of  the  Recruit  Training 
Command,  Naval  Training  Center,  San  Diego,  who  performed  the  nec- 
essary arrangements  relative  to  facilities,  scheduling,  and  the  like. 
The  major  contributions  during  the  test  administrative  phases  of  CPO 
Robert  G.  Semperger  and  CPO  William  Parks  of  the  Apprenticeship 
Training  Department,  and  Mr.  W.  Rick  Leahy  and  Dr.  Robert  M. 
Voytas,  both  of  Applied  Psychological  Services,  are  acknowledged. 
These  persons  also  made  a number  of  suggestions  for  increasing 
the  effectiveness  and  efficiency  of  the  methods  employed. 


J.  J.  CLARKIN 
Commanding  Officer 
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SUMMARY 


Problem 

Current  Navy  Recruit  classification  methods  are  based  on 
paper-and -pencil  testing  techniques.  There  is  no  available  method 
for  reclassifying  persons  who  are  originally  assigned  to  "general 
service"  but  who  warrant  reclassification.  The  result  is  often  im- 
proper and  noncost/effective  personnel  utilization. 

Objective 

The  purpose  of  this  effort  was  to  develop  a method  for  re- 
classifying Navy  personnel  who  did  not  qualify  for  assignment  to  an 
"A"  school  on  the  basis  of  aptitude  scores  derived  from  a conven- 
tional classification  battery. 

Approach 

Two  recent  evaluative  developments --the  assessment  center 
approach  and  an  exercise  development  model  which  is  based  on  the 
logic  that  a person  who  demonstrates  the  ability  to  learn  a sample 
of  a job  will  be  able  (given  appropriate  training)  to  learn  the  total 
job--were  woven  into  a classification/  reclassification  scheme.  The 
scheme  is  based  on  "hands  on"  training  procedures  and  work  sample 
performance  evaluative  methods.  Navy  jobs  were  analyzed,  and  a 
set  of  exercises,  based  on  the  two  concepts,  was  developed.  These 
exercises  were  then  incorporated  into  a traditional  management  as- 
sessment center  paradigm  and  the  methods  tried  on  140  enlisted 
persons  who  were  previously  judged  to  be  not  qualified  for  "A" 
school  training. 


Findings 

Of  the  140  men  involved,  76  were  considered,  as  the  result 
of  application  of  the  methods  employed,  to  possess  capability  for 
adequate  performance  in  one  of  the  following  specialties:  Postal 
Clerk,  Storekeeper,  Yeoman,  Signalman,  Machinist's  Mate,  Hospi- 
tal Corpsman,  Engineman,  and  Electrician's  Mate.  The  reliability 
of  the  exercises  and  of  the  subsequent  classification  decisions  was 
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very  acceptable.  Consistent  classification  policies  were  followed 
by  the  various  staff  members  from  whom  the  classification  recom- 
mendations evolved.  I he  individuals  assessed  considered  the  multi- 
ple assessment  methods  employed  to  be  fairer  than  and  preferable 
to  the  usual  methods  which  are  hinged  to  paper-and-pencil  evaluative 
techniques. 


Conclusions 


The  approach  to  classification/ reclassification  here  devel- 
oped seems  entirely  workable,  useful,  and  within  acceptable  stand- 
ards of  fairness.  At  least  from  the  points  of  view  investigated,  the 
approach  also  possesses  adequate  psychometric  properties.  The  ap- 
pioach  can  identify  persons  who  seem  to  possess  capability  which  is 
not  identified  by  the  usual  classification  methods. 
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INTRODUCTION 


Problem 


The  Navy  has  traditionally  based  classification  decisions 
largely  upon  paper  and  pencil  tests  which  have  been  validated  against 
"A"  school  grades.  The  problem  is  that  direct  measures  of  an  in- 
dividual's ability  to  perform  the  requirements  of  a job  are  not  direct- 
ly assessed.  Accordingly,  a large  number  of  recruits  who  are  unable 
to  qualify  for  formal  "A"  school  training  are  assigned  directly  to  the 
Fleet  where  they  must  rely  upon  on-the-job  training  and  personal  ini- 
tiative to  compete  successfully  for  promotion  with  their  "A"  school 
trained  counterparts.  Little  is  known  about  the  potential  of  general 
service  personnel  to  perform  in  a formal  school  setting  because  of 
their  inability  to  satisfy  the  entrance  requirements  as  determined 
by  test  results.  In  the  present  effort  representative  performance 
job  samples  were  developed  to  investigate  their  utility  as  supple- 
mental measures  in  assessing  "general  service"  personnel  for  re- 
classification and  possible  "A"  school  assignment. 


Purpose 


The  overall  goal  of  the  present  program  was  to  develop  and 
demonstrate  a method  for  reclassification  of  Navy  personnel  who.-  (a) 
are  originally  considered  unsuitable  for  "A"  school  training,  or  (b)  de- 
sire reclassification  because  they  are  not  satisfied  with  their  original 
classification.  The  need  for  improved  methods  for  personnel  reas- 
sessment stems  from  the  fact  that,  at  present,  the  Navy  possesses  no 
fair,  objective,  valid,  and  standardized  technique  for  reassessing  the 
enlisted  personnel  after  they  are  initially  classified.  At  the  time  of  the 
study,  the  classification  method  depended  on  the  use  of  the  Navy's  "bas- 
ic battery. " This  battery  consists  of  a general  aptitude  test  (GCT),  a 
mechanical  aptitude  test,  an  arithmetic  test,  an  electronic  aptitude 
test,  a shop  practices  test,  and  a clerical  test.1  On  the  basis  of 
scores  on  these  instruments  plus  interest  information,  a recruit  is 
classified  and  assigned  to  an  "A"  school  or  directly  to  the  Fleet.  Mis- 
classifications,  expecially  in  the  case  of  those  recruits  sent  directly 
to  the  Fleet,  are  costly  to  the  Navy  and  to  the  general  society.  Such 
misclassified  persons  are  apt  to  advance  only  slowly  in  the  Fleet  and 
are  more  apt,  as  a consequence,  to  become  disciplinary  problen  s. 
Moreover,  and  of  greater  importance,  due  to  misclassification.  a sailor 
may  be  classified/ assigned  so  that  he  contributes  less  than  his  full  potential. 

iAs  of  January  1976  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  has  replaced  the  Basic  Test  Battery  (BTB)  as  the  classifica- 
tion instrument.  . 


The  method  developed  and  evaluated  here  is  called  the 
Technical  Classification  Assessment  Center  (TCAC)  approach.  It 
emphasized  exercises  which  allow  the  individual  to  learn  and  per 
form  tasks  that  simulate  likely  on-the-job  learning  and  perform- 
ance. The  basic  concepts  underlying  this  method  were: 

• The  individual  deals  with  any  task  by  apply- 
ing in  an  integrated  fashion  his  total  comple- 
ment of  aptitudes  and  abilities  (Rundquist, 

1969). 

• Results  from  exercises  which  allow  the  indi- 
vidual to  demonstrate  how  well  he  can  learn 
aspects  of  a job  can  provide  a basis  for  pre- 
dicting performance  on  the  whole  job  (Siegel 
iv  Bergman.  1975). 

• The  assessment  center  performance  test  ap- 
proach, in  which  several  test  administrators 
representing  possible  diverse  points  of  view 
observe  an  individual  as  he  performs  job  re- 
lated tasks,  has  promise  for  technical  level 
jobs  (Bray  & Moses,  1972). 


Assessment  Centers 

Current  use  of  assessment  centers  is  limited  almost  entire- 
ly to  managerial  level  personnel.  The  extension  of  the  assessment 
center  approach  to  technical  level  jobs  represents  an  elaboration  of 
the  assessment  center  concept,  as  originally  developed.  However, 
the  extension  of  the  assessment  center  approach  to  nonmanagerial 
jobs  was  previously  suggested  as  a possible  area  of  investigation 
by  Bray  and  Moses  (1972  ). 

Several  reviews  of  current  assessment  center  practices 
are  available  (MacKinnon,  1975;  Howard,  1974;  Huck,  1973;  Bray 
& Grant,  1966).  The  assessment  center  approach  treats  each  in- 
dividual as  a whole  person  rather  than  as  a sum  of  specific  abilities 
and  aptitudes.  The  typical  assessment  center  runs  for  one  to  three 
days  and  involves  two  to  five  administrators  (assessors)  and  six  as- 
sessees.  The  goal  is  to  predict  future  managerial  success.  Typi- 
cal measurements  and  tasks  include: 
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• Inbasket  tests --memoranda,  letters,  and 
notes  are  presented,  and  the  person  being 
assessed  indicates  (usually  in  writing)  how 
he  would  handle  each  of  these 

• Group  discussion --usually  a leaderless  dis- 
cussion possessing  the  goal  of  formulating  a 
solution  to  a specific  problem 

• Objective  tests--including  paper  and  pencil 
aptitude  and  ability  tests 

• Interview 

• Other  techniques--such  as  projective  tests 
and  role  playing  situations 

The  assessment  center  approach  represents  a consolidation 
of  clinical,  gestalt,  and  behavioristic  evaluation.  The  clinical  ap- 
proach is  evident  in  the  judgments  involved  in  formulating  assess- 
ment center  recommendations  based  on  data  from  and  observation 
of  performance  of  the  above  and  similar  tasks  on  variables  such  as: 
aggressiveness,  energy  level,  interpersonal  contact,  self  confidence, 
sensitivity,  dependence  on  others,  work  motivation,  and  adaptability 
(Hinrichs,  1969;  Bray  & Grant,  1966).  The  gestalt  aspect  is  seen  in 
the  assessment  center's  final  rating  of  each  individual  as  a whole  per- 
son (in  terms  of  whether  or  not  or  how  well  the  person  will  succeed) 
rather  than  a rating  based  on  a statistically  weighted  sum  of  various 
scores  (Bray  & Campbell,  1968).  The  behavioristic  influence  is  re- 
flected in  the  emphasis  on  rating  of  actual  behaviors  in  (largely  sim- 
ulated) real  life  situations. 

There  is  also  an  apparent  psychometric  influence.  The  psy- 
chometric influence  is  seen  in  the  fairly  numerous  attempts  to  meas- 
ure the  reliability  and  the  validity  of  the  assessment  center  judgments 
(e.  g. , Bray  & Grant,  1966;  Bray  & Campbell,  1968;  Hinrichs,  1969; 
and  Greenwood  & McNamara,  1967,  1969). 

The  present  program  represents,  to  our  knowledge,  the  first 
extension  of  the  assessment  center  approach  to  technical  jobs  (such 
as  the  Navy  machinist's  mate,  postal  clerk,  and  storekeeper).  All 
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of  the  major  aspects  of  the  assessment  cei  ter  approach  were  pre- 
served in  the  current  TCAC.  However,  the  evaluative  instruments 
often  used  in  assessment  centers  are  designed  for  managerial  po- 
sitions. Tests  of  this  nature  (e. g. , inbasket  tests)  are  not  appro- 
priate for  technical  level  jobs.  In  the  present  TCAC,  such  assess- 
ment tools  were  replaced  by  exercises  related  to  technical  jobs. 
Following  usual  assessment  center  philosophy,  the  assessee  is  re- 
quired, in  dealing  with  these  exercises,  to  draw  on  all  of  his  abil- 
ities and  aptitudes  in  an  integrated  fashion. 


The  validity  of  the  assessment  center  approach  has  been  usu- 
ally evaluated  in  terms  of  prediction  of  future  managerial  success. 

In  this  essential  respect,  the  assessment  center  approach  seems  to 
work.  Predictive  validity  correlation  coefficients  of  .40  and  higher 
are  not  uncommon  in  the  literature  (MacKinnon,  1975;  Ash  & Kroeker, 
1975;  Mitchel,  1975;  Howard,  1974;  Bray  & Moses,  1972;  and  Aol- 
lowick  & McNamara,  1969).  Prior  to  the  availability  of  validity  data, 
an  assessment  center  may  be  evaluated  on  the  basis  of  descriptive 
statistics  relative  to  the  assessment  situations  employed  and  the  judg- 
ments of  the  assessors. 


Evaluation  of  an  assessment  center  should  also  include  con- 
sideration of  several  aspects  of  the  practical  operation  of  the  center. 
The  assessment  center  should  be  easily  administered  and  should  not 
unduly  disrupt  the  ongoing  operation  of  the  parent  organization.  The 
assessment  experience  should  be  a profitable  one  for  the  assessees, 
and  the  experience  should  possess  face  validity.  The  public  relations 
function  that  face  validity  serves  (APA,  1974)  is  becoming  more  im- 
portant as  the  general  public  becomes  more  sophisticated  and  more 
wary  concerning  psychological  testing. 


As  the  result  of  a recent  Nebraska  Court  Decision  (Anony- 
mous, 1976),  several  other  important  criteria  have  emerged; 


• Administrators  must  be  adequately  trained 
in  how  to  conduct  the  assessment  center. 


The  assessment  materials  should  be  based 
on  a job  analysis. 


• The  exercises  used  must  be  appropriate. 
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• There  must  be  an  adequate  number  of 
exercises. 

• Assessors  should  not  know  candidates  be- 
fore the  assessment  and  assessors  and 
candidates  should  be  randomly  assigned. 

• The  center  should  be  administered  in  a 
professional  manner. 

• Assessor  effectiveness  and  reliability 
should  be  continuously  monitored. 

In  planning  and  implementing  any  testing  program,  consid- 
eration of  certain  ethical  questions  is  required.  Basic  ethical 
guidelines  for  assessment  center  operations  have  been  developed 
and  are  summarized  by  Moses  et  al.  (1975).  They  discuss  five 
areas  that  require  consideration:  organizational  support  for  assess- 
ment operations,  assessor  training,  informed  consent  on  the  part 
of  participants,  use  of  assessment  center  data,  and  validation  is- 
sues. Although  this  statement  on  ethical  principles  was  not  avail- 
able at  the  onset  of  this  project,  these  areas  were  considered  and 
in  all  cases  the  recommended  minimum  standards  were  met  or  sur- 
passed. 

Job  Sample  Performance  Tests 

Performance  tests  in  general  and  particularly  the  job  sam- 
ple variety  have  become  the  object  of  increased  recent  interest 
for  two  reasons.  First,  performance  tests  may  yield  an  improve- 
ment over  the  modest  levels  of  prediction  of  job  success  which  re- 
sult from  the  use  of  the  paper-and-pencil  tests.  Second,  job  sam- 
ple tests  seem  to  conform  to  the  job  relatedness  criterion  of  recent 
court  decisions  (Ash  & Kroeker,  1975;  O'Leary,  1973). 

The  idea  of  using  a sample  of  actual  work  behavior  as  a pre- 
dictor of  future  job  success  is  a relatively  old  one.  The  use  of  job 
sample  tests  in  clerical  settings  dates  back  at  least  to  1933  (Albright, 
Glennon,  & Smith,  1963).  As  long  ago  as  1947,  a text  describing 
job  sample  performance  tests  described  in  detail  most,  if  not  all, 
of  the  techniques  in  use  today  (Adkins,  Primoff,  McAdoo,  Bridges, 
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& Forer,  1947).  Through  the  years  these  techniques  have  been 
applied  often  in  both  military  (e.  g. , Greer,  Pearson,  & Havron, 

1957;  Jensen,  Hill,  Siegel,  & Courtney,  1954;  Siegel  & Courtney, 
1953)  and  civilian  (e.  g. , Campion,  1972;  Tiffin,  1952)  settings. 

Performance  tests  are  usually  considered  in  contrast  to 
paper  and  pencil  tests.  One  class  of  performance  tests  attempts 
to  measure  mechanical  or  psychemotor  abilities  or  aptitudes  in 
isolation.  Tests  of  this  type  are  of  little  interest  in  the  present 
context.  A second  class  of  performance  test,  the  job  (or  work) 
sample  performance  test,  requires  the  individual  to  perform  a 
task  which  is  similar  or  identical  to  one  performed  on  the  job. 

Two  different  philosophies  underlie  the  use  of  these  two  types  of 
performance  tests.  The  tests  of  specific  psychomotor  abilities  are 
based  on  a specific  abilities  philosophy.  This  philosophy  holds  that 
final  job  performance  can  be  predicted  based  on  measures  ot  the 
independent  unitary  abilities  required  by  the  job.  Such  specific 
abilities  may  include;  choice  reaction  time,  wrist-finger  speed, 
manual  dexterity,  and  speed  of  limb  movement  (McCormick  & 

Tiffin,  1975).  The  tests  that  consist  of  job  samples  are  based 
on  an  approach  which  usually  emphasizes  the  overall  capabilities 
of  a person.  Each  person  is  assumed  to  bring  to  a task  a differ- 
ent complement  of  abilities.  These  are  applied  to  the  performance 
of  the  task  in  unique  ways.  Two  people  may  perform  well  based  on 
tw'o  different  complements  of  abilities.  Rundquist  (1969)  made  this 
point.  Referring  to  aptitude  scores  or  actual  job  performance,  he 
wrote,  ". . . phenotypic  score  similarities  are  no  guarantee  of  sim- 
ilarities in  the  basic  processes  involved.  " Rundquist  also  suggested 
that  the  failure  to  consider  the  dynamic  interrelationships  of  abilities 
in  each  person  may  have  limited  current  methods  for  developing  pre- 
dictors to  their  current  low  levels  of  validity.  It  is  likely,  both  from 
the  perception  of  individuals  and  in  reality,  that  individuals  deal  with 
real  life  situations  by  applying  their  resources  as  adaptively  as  pos- 
sible. This  may  differ  from  person  to  person;  actual  on-the-job  per- 
formance may  not  be  based  on  the  same  ability  or  combination  of 
abilities  for  every  person  (Rundquist,  1969).  Thus,  job  perform- 
ance may  not  be  well  predicted  by  one  aptitude  or  ability  score  or 
even  by  a linear  combination  of  such  scores. 
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The  performance  oriented  tests  of  the  present  program 
which  simulate  real  on-the-job  activities  allow  the  individual  to 
demonstrate  how  effectively  he  can  apply  his  complement  of  apti- 
tudes and  skills  to  complex  situations  akin  to  those  encountered 
in  real  life  by  Navy  enlisted  personnel.  In  using  this  approach, 
one  tests  the  ability  of  the  individual  to  perform  an  actual  or  simu- 
lated sample  of  the  actual  job.  There  is  some  evidence  that  this 
approach  is  valid  for  technical  level  jobs  (Campion,  1972;  McCor- 
mick & Tiffin,  1975).  Campion  (1972)  compared  job  sample  tests 
with  several  paper  and  pencil  tests  (including  tests  of  mechanical 
comprehension,  intelligence,  and  numerical  and  clerical  ability). 
Correlations  between  job  sample  predictors  and  criterion  variables 
were  .66,  .42,  and  .46,  while  correlations  between  papemand-pen- 
cil  test  scores  and  the  criterion  variables  were  low,  ranging  from 
-.  23  to  . 08. 

Job  Learning  Approach  to  Testing 


An  approach  to  performance  tests,  recently  developed 
through  a systematic  set  of  studies  performed  at  Applied  Psycho- 
logical Services,  is  based  on  the  idea  that  if  a man  demonstrates 
the  ability  to  learn  a representative  sample  of  a job,  then  he  can 
be  expected  to  be  able  to  learn  the  whole  job  (Siegel  & Bergman, 
1975;  Siegel  & Leahy,  1974;  Siegel,  Bergman,  & Lambert,  1973; 
Siegel  & Bergman,  1972).  In  this  approach,  each  evaluative  situ- 
ation involves  two  phases;  (1)  a training  phase,  and  (2)  an  evalua- 
tion phase.  An  example  of  this  approach,  described  by  Siegel  & 
Bergman  (1975),  includes  a detailed  description  of  how  to  start  up 
and  shut  down  a motor  and  pump  apparatus.  This  procedure  in- 
volves 33  steps  including  several  safety  precautions.  Each  ex- 
aminee is  then  given  an  opportunity  for  supervised  "hands  on" 
practice  of  this  procedure.  After  this  practice  session,  a per- 
formance test  is  administered.  Other  tests  of  this  type  described 
by  Siegel  and  Bergman  include;  tool  identification  and  use,  gasket 
cutting  and  meter  reading,  troubleshooting  on  a pumping  system, 
and  assembly  of  a gate  valve.  These  tests  were  developed  to  as- 
sess promise  in  the  Navy  machinist's  mate  career  field. 
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This  approach  to  performance  testing  has  been  termed 
the  miniature  job  training  and  evaluation  approach.  An  earlier 
related  approach  was  used  by  Lawshe  and  Tiffin  in  the  develop- 
ment of  the  Purdue  Mechanical  Adaptability  Test.  This  test, 
described  by  McCormick  & Tiffin  (1975),  was  developed  on  the 
basis  that; 

...there  was  reason  to  believe  from  a previous 
study  that,  other  things  being  equal  (emphasis 
added),  those  persons  who  have  most  profited  in 
knowledge  from  previous  mechanical  experiences 
may  do  better  on  mechanical  jobs  than  those 
persons  who  have  not  so  profited  (pp.  143-144). 

In  concordance  with  this,  Siegel  and  Bergman  (1975)  suggested 
that  the  miniature  task  training  part  of  their  approach  gives  all 
the  individuals  tested  a fair  chance  to  do  well  on  the  test  in  a way 
that  a performance  test  without  a learning  phase  does  not.  The 
effects  of  differences  in  exposure  to  mechanical  devices  in  gen- 
eral and  specifically  to  the  job  sample  considered  are  thought  to 
be  controlled  due  to  the  equalizing  effect  of  the  training  phase. 

Siegel  and  Bergman  (1975)  constructed  their  miniature 
job  training  and  aptitude  tests  so  as  to  minimize  emphasis  on 
the  ability  to  read  and  write.  Ash  and  Kroeker  (1975)  cautioned 
against  the  use  of  tests  which  have  a higher  reading  level  than 
that  required  by  the  job.  Such  tests  may  be  inherently  biased 
against  certain  classes  of  people  who,  for  reasons  associated 
with  their  culture  or  socioeconomic  class,  have  had  less  formal 
education,  or  less  successful  formal  education,  than  other  classes. 
For  a motivated  person,  formal  education,  which  yields  a facility 
with  written  English,  may  have  little  to  do  with  job  success  in 
many  jobs. 

The  job  learning  approach  as  used  with  low  aptitude  Navy 
enlisted  personnel  by  Siegel  and  Bergman  (1972)  was  successful 
in  several  respects.  Predictive  validity  multiple  correlation  co- 
efficients of  . 15  to  . 46  were  obtained.  This  compares  favorably 
with  the  predictive  validity  of  the  Navy  test  scores  in  this  situa- 
tion. Siegel  and  Leahy  (1974)  extended  these  findings  with  reports 
of  subjective  evaluations  of  the  sailors  selected  with  the  new  mini- 
ature training  and  evaluation  method.  Supervisors  of  the  men 
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recommended  indicated  that,  if  given  a choice,  in  79  percent 
of  the  cases  they  would  choose  the  recommended  man  as  a sub- 
ordinate. Siegel  and  Leahy  also  reported  that  despite  careful 
analysis,  no  race  related  differences  in  performance  or  these 
tests  were  detected.  Siegel  and  Bergman  (1975)  concluded  that 
this  type  of  test  has  merit  and  promise  in  three  areas:  (a)  pre- 
dicting performance  of  "low  aptitude"  applicants,  (b)  extension 
to  other  less  mechanical  jobs,  and  (c)  valid  testing  which  is  al- 
so fair  to  minority  groups.  The  present  study  adapted  several 
of  the  miniature  training  and  evaluation  tests  developed  by  Siegel 
and  his  associates  at  Applied  Psychological  Services.  The  ap- 
proach is  extended  in  the  present  program  to  other  technical  lev- 
el jobs,  including  some  jobs  that  are  low  on  mechanical  ability 
requirements. 


METHODS 
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The  overall  purpose  of  the  present  program  was  to  estab- 
lish a classification/ reclassification  technique  for  non  "A"  school 
listed  personnel  in  the  Navy.  To  this  end,  the  assessment  center 
concept  was  melded  with  the  miniature  job  learning  and  aptitude 
test  approach  to  yield  an  articulated  scheme  possessing  the  merits 
of  both  concepts.  To  achieve  this  combination,  several  sequential 
steps  were  required: 

a.  Job  analysis-performance  of  a three  dimen- 
sional conceptual  analysis  of  NAVPERS  18068C 
(Manual  of  Qualifications  for  Advancement  in 
Rating)  and  a choice  of  target  Navy  career 
fields  for  consideration. 

b.  Exercise  development-development  of  the  spe- 
cific exercises  to  be  included  in  the  TCAC  on 

the  basis  of  the  job  analysis  and  Guilford's  Struc- 
ture-of-Intellect  model. 

c.  Administrative  methods--development  of  a com- 
plete administrator's  manual,  training  the  ad- 
ministrators, and  development  of  decision  mak- 
ing procedures. 

d.  Evaluation  plans--development  of  a systematic 
procedure  for  evaluating  the  adequacy  of  the 
TCAC  including  the  exercises,  per  se,  the 
recommendations,  and  the  reliability  of  the 
administrators. 

Each  of  these  developments,  along  with  their  rationale,  is 
described  in  subsequent  sections  of  this  chapter. 


Preceding  page  blank 
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.Job  Analysis 


The  Manual  of  Qualifications  for  Advancement  in  Ratings 
(NAVPERS  18068C)  was  taken  as  the  basic  source  of  information 
for  the  job  analysis.  This  .Manual  contains  detailed,  updated  lists 
of  the  tasks  performed  by  personnel  in  all  Navy  specialties.  The 
tasks  required  for  a promotion  to  the  E4  level  were  chosen  for 
analysis  as  this  level  is  usually  reached  within  the  original  enlist- 
ment period  (four  years)  of  typical  enlisted  Navy  personnel.  The 
importance  of  a close  correspondence  between  the  level  of  the 
individuals  tested  and  the  level  of  jobs  considered  in  validation 
studies  was  emphasized  by  the  Supreme  Court  (Lifter,  1976). 

I he  task  statements  in  NAVPERS  18068C  are  different  n 
one  respect  from  many  other  job  analyses.  The  requirements 
given  in  NA\  PERS  18068C  apply  to  all  enlisted  men  in  the  speci- 
fied rating.  The  actual  jobs  men  in  these  ratings  perform  may 
vary  from  ship  to  ship  and  from  assignment  to  assignment.  How- 
ever, all  men  in  a given  rating  are  expected  to  be  able  to  perform 
the  tasks  listed  for  that  rating.  The  use  of  NAVPERS  18068C  was 
dictated  by  the  practical  impossibility  of  developing  a workable 
system  with  considerably  finer  job  discriminations  than  those  al- 
ready in  use  by  the  Navy  (with  nine  levels  of  each  of  over  80  jobs). 
The  use  of  NAVPERS  18068C  had  the  added  virtue  of  keeping  the 
present  study  in  step  with  the  most  current  Navy  statement  of  re- 
quirements of  the  various  rates  and  ratings. 

Analysis  of  NAVPERS  18068C  identified  three  dimensions 
which  appeared  to  be  relevant  to  all  Navy  ratings:  (a)  learning, 

(b)  psychophysical  and  motor,  and  (c)  social-motivational.  This 
structure,  in  a sense,  parallels  the  classification  of  aptitude  tests 
developed  by  Ghiselli  (1966,  1 973).  Ghiselli  classified  the  various 
types  of  aptitude  tests  into  five  categories:  (a)  intellectual  abilities, 
(b)  spatial  and  mechanical  abilities,  (c)  perceptual  accuracy,  (d) 
motor  abilities,  and  (e)  personality  traits.  The  dimensionaliza- 
tion  used  in  the  present  study  effectively  consolidates  Ghiselli :s 
three  middle  categories  into  one  category.  These  three  dimen- 
sions were,  accordingly,  considered  to  cover  comprehensively 
most  skills,  abilities,  and  aptitudes  requisite  for  on-the-job  suc- 
cess for  enlisted  personnel  in  the  Navy. 

A sample  page  from  NAVPERS  18068C  is  presented  in  Fig- 
ure 1. 
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QUALIFICATIONS  FOR  ADVANCEMENT  Required  for 


QUALIFICATIONS  FOR  ADVANCEMENT  Required  for 

Advancement  to 

B.  OFFICE  PROCEDURES  - Continued  YN 

1 . 00  Practical  Factors  - Continued 

.04  Meet  enlisted,  officer,  and  civilian  personnel  as  a receptionist E-4 

. 05  Speak  clearly  and  distinctly E-4 

.08  Demonstrate  courtesy,  tact,  and  good  judgment  when  handling 

telephone  communications E-4 

.40  Organize  and  maintain  files,  including  files  of  directives  and 

correspondence,  and  cross  reference  sheets  and  tickler  files  .......  E-5 

2.00  Knowledge  Factors 

.01  Proper  English  grammar  and  punctuation  E-4 

.40  Regulations  concerning  disposal,  stowage,  and  transmission  of 

obsolete  files  and  records  E-5 


C.  EQUIPMENT  OPERATION 

1.00  Practical  Factors 


.03  Operate  office  duplicating  equipment  E-4 

.39  Typewrite  for  5 minutes:  (See  Performance  Test  Instructions) 

a.  At  30  words  per  minute  E-4 

b.  At  40  words  per  minute  E-5 

e.  At  50  words  per  minute E-6 


2.00  Knowledge  Factors 

None  in  addition  to  those  implied  in  the  above  Practical  Factors. 

D.  REPORTS,  PUBLICATIONS,  AND  RECORDS 

1.00  Practical  Factors 


.01  Transcribe  officer  message  orders,  using  current  instructions E-4 

.02  Maintain  publications E-4 

.40  Maintain  officer  personnel  diaries  in  accordance  with  Manpower  and 

Personnel  Management  Information  System  Manual  (MAPMISMAN)  . . . E-5 

.41  Maintain  officer  service  records E-5 

.42  Verify  the  Officer  Distribution  Control  Reoort  (ODCR)  and  other 

related  reports  as  required  by  MAPMISMAN E-5 

.43  Prepare  Manpower  Authorization  Requests  E-5 

.GO  Supervise  the  procurement,  maintenance,  stowage,  issuance,  and 

custody  of  official  publications E-6 


2.00  Knowledge  Factors 

.01  General  content  and  use  of  standard  publications  pertaining  to 
personnel  and  general  administration,  including  the  following: 


a.  U.S.  Navy  Regulations E-4 

b.  Bureau  of  Naval  Personnel  Manual E-4 


Figure  1.  Sample  NAVPERS  18063C  page. 


13 


A working  definition  of  each  of  these  dimensions  was  de- 
veloped in  order  to  evaluate  the  various  Navy  career  fields.  These 
definitions  are: 

• Learning-requirements  for  acquiring  complex 
knowledges  and  skills  and  for  integrating  these 
into  purposeful  activity. 

• Psychophysical  and  motor-requirements  for 
fine  discriminations,  complex  psychomotor  co- 
ordination, attentiveness,  and  precision,  and  for 
integrating  these  into  a rapid  response  possibil- 
ity under  stressful  and/or  time  limited  and  time 
critical  conditions. 

• Social/ motivational  — requirements  for  demon- 
strating such  characteristics  as  initiative,  leader- 
ship behavior,  social  interactive  skill,  and  per- 
sonal characteristics  consistent  with  mission 
goals. 


Two  independent  raters  evaluated  the  knowledge  and  skill 
requirements  of  the  general  ratings  described  in  NAVPERS  18068C 
(with  some  exceptions,  such  as  the  aviation  group).  Based  on  con- 
sideration of  the  skills  and  knowledges  required  for  promotion  to  the 
E4  level,  each  judge  rated  each  Navy  rating  relative  to  each  of  the 
three  conceptual  categories.  The  judgments  for  each  category  were 
made  on  a four  level  scale  rating  from  "none  of  this  factor  is  re- 
quired for  adequate  job  performance"  through  "a  great  deal  of  this 
factor  is  required  for  adequate  job  performance.  " 

After  the  two  raters  independently  completed  their  ratings, 
any  differences  between  the  two  were  discussed.  Based  on  con- 
sideration of  the  details  for  each  rating,  as  given  in  NAVPERS 
18068C,  any  disagreements  in  the  evaluations  were  resolved.  The 
resulting  final  evaluations  are  given  in  Table  1. 
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Requirements  of  Various  Navy  Ratings  Along  the  Learning,  Psycho- 
physical and  Motor,  and  Social/ Motivational  Dimensions 


GROUP  I DECK 
Boatswain's  Kate  (BM) 

Electronics  Warfare  Tecnr.icia:.  (EW> 

Quart ermas  t er  < Q,M ) 

Operations  Specia_ist  (OS) 

Signalman  (S!'i 
Sonar  Technician  (ST' 

Master  at  Arms 

GROUP  II  ORDINANCE 

Eire  Control  Technician  (FT) 

Gunner's  Nate  (3M } 

Minenar.  (MK> 

Missile  Technician  (MT) 

Torpedoman ' e Ma te  ( TM ) 

GROUP  III  ELECTRONICS 

Lata  Systems  Technician  ( OS ) 

Electronic:.  Technician  (ET 

GROUP  IV  PRECISION  EQUIPMENT 

Instrumentmar.  ( IK  • 

Opticalmar.  (DM) 

GROUP  V ADMINISTRATIVE  AND  CLEKiC.’L 

C orm  i sar  ymar.  ( CS  » 

Communications  Technician  t CT ) 

Cat-  rocessing  Technician  (OP) 
Disbursing  Clerk  ( DK ' 

Journalist  (JC* 

Personnelman  ( PU ’ 

Postal  Cierr  (PC ) 

Radictar.  (EM  > 

Shit's  Servicema-  (SH) 

Storekeeper  (Sr) 

Y ee  r.ar.  ( YN  '* 

.tewarc  ( SD ) 

GROUP  VI  MISCELLANEOUS 

Illustrator  Draftsman  (CM’ 

Litnccrapner  (LI) 

Musician  (MV) 

GROUP  VII  ENGINEERING  AND  HULL 

Bolder  Technician  (3T) 

Electrician's  Kate  (EK> 

Engineman  (EN ' 

Hull  Maintenance  Tecnnician  (H7) 

Interior  lemmuni cat  ions  Electriciat  (IC) 
Kachinerv  Fetairmar.  (MF> 

Machinist's  Kate  (MM) 

Molder  (ML) 

Patternmaker  ( PM ) 

Gas  Tjrbine  System  Tecnnician  (GS) 

GROUP  VIII  CONSTRUCTION 

Builder  (BU) 

Construction  Electrician  (CE) 
Construction  Mechanic  (CM) 

Engineering  Aid  (EA) 

Equipment  Dperaror  (EO) 

Steel  Worder  l SW ' 

Utilitiesman  (UT> 

GROL.  X MEDICAL 

Hospital  iorpsmar  (HM) 

GROUP  XI  DENTAL 

Dental  Technician  (DT5 


Learning  Psychophysical /Kotos  Social /Motivation*. 


Key  to  scale  values: 

1=  little  of  this  dimension  required  for  adequate  job  performance 
2=  some  of  this  dimension  required  for  adequate  job  performance 
3=  considerable  amount  of  this  dimension  required  for  adequate  job  performance 
4=  a great  deal  of  this  dimension  required  for  adequate  job  performance 
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Exercise  Development 


The  design,  form,  and  content  of  the  exercises  included  in 
the  TCAC  was  guided  by  the  analysis  of  NAVSHIPS  18068C,  which 
resulted  in  the  identification  of  the  three  dimensions:  learning 
psychophysical  and  motor,  and  social/ motivational,  and  by  the 
subsequent  selection  of  target  ratings. 

However,  it  seemed  clear,  on  the  basis  of  the  analytic 
method  used  and  on  the  literature,  that  each  of  the  three  dimen- 
sions is  not,  in  fact,  unidimensional.  Accordingly,  an  effort  was 
made  to  assure  that  the  exercise  battery  sampled  as  many  of  the 
individual  aspects  of  these  dimensions  as  possible.  To  this  end, 
an  empirically  derived  theory  concerning  the  basic  factors  of  in- 
tellective function  (Guilford  & Hoepfner,  1971)  was  used  to  help 
guide  the  design  of  the  individual  exercises  and  to  help  assure  a 
balanced  test  battery. 

Guilford's  Structure-of-Intellect  model  of  intellective  func- 
tioning (Guilford  & Hoepfner,  1971)  was  used  as  a guide  in  the  de- 
sign and  development  of  the  exercises  used  to  evaluate  abilities  in 
the  learning  and  the  psychophysical  and  motor  dimensions.  Exer- 
cises were  selected  and  developed  so  that  each  combination  of 
Guilford's  intellective  operations  (cognition,  memory,  and  evalu- 
ation) and  intellective  contents  (figural,  symbolic  and  semantic) 
would  be  required  by  at  least  one  test.  While  the  other  parameter  s 
of  Guilford's  theory  seem  to  be  sampled  by  the  exercises  developed, 
emphasis  in  this  developmental  stage  was  placed  on  the  parameters 
mentioned.  The  tests  developed  to  assess  the  learning  and  the  psy- 
chophysical and  motor  dimensions,  together  with  the  combination  of 
the  Guilford  parameters  sampled  by  these  tests,  are  given  in  Table  3. 
The  specifics  of  the  various  exercises  are  presented  later  in  this 
chapter. 

Of  the  six  exercises  included  in  Table  3,  the  Conceptual  In- 
tegration/Application, Tool  and  Object  Nomenclature,  Use,  and  Rec- 
ognition, Dual  Tasks,  and  Coordinative  Speed  and  Accuracy  exer- 
cises are  based  on  the  job  learning  concept.  The  Inspection/ Sort 
exercise  does  not  include  a learning  phase. 


Table  3 


Structure-of-Intellect  Analysis  of  the  Exercises  Used  to  Assess 
the  Learning  and  the  Psychophysical  and  Motor  Dimensions 


Psychophysical  and  Motor 


Contents 

Operations 

Cognition 

Memory 

Evaluation 


Learning 

Symbolic 

Semantic 

4,6 

1,6 

1,6 

1,6 

1,3,4 

1 

Key  to  tests 

1=  Conceptual  Integration/Application  (troubleshooting) 
2=  Coordinative  Speed  and  Accuracy  (wiring) 

3=  Dual  Task  (control  watch) 

4=  Dual  Task  (pipe  assembly) 

5=  Inspection/Sort 

6=  Tool  Object  and  Nomenclature,  Use,  and  Recognition 


Additionally,  several  exercises  were  designed  to  measure 
the  social/ motivational  dimension.  The  performance  tests  devel- 
oped to  sample  this  dimension  involve  an  actual  group  task  (the 
Social  Interactive  Evaluation),  a test  of  honesty  (the  Reliability 
Evaluation),  and  a game  situation  in  which  some  intrinsic  aspects 
of  motivation  are  believed  to  be  evidenced  (the  Level  of  Aspiration 
Evaluation).  Finally,  a semistructured  interview  was  developed. 
The  interview  was  designed  to  approach  topics  in  three  areas; 

(a'  interest  and  background  (e.  g. , areas,  tasks,  and  jobs  the  indi- 
vidual likes  to  deal  in  or  is  drawn  to),  (b)  motivation  (e.  g. , does 
the  individual  being  assessed  possess  goals  and  is  he  driven  to  at- 
tain these  goals,  ability  of  the  individual  being  assessed  to  recon- 
cile personal  needs  with  those  of  the  Navy),  and  (c)  general  capac- 
ity (e.  g.  , common  sense,  alertness  level,  fund  of  general  know- 
ledge). Additionally,  topics  indicated  by  performance  on  the  various 
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evaluative  situations  during  the  day  were  broached  in  the  interview. 
The  desired  end  product  from  the  interview  was  an  evaluation  of 
each  individual  on  each  of  several  aspects  (such  as  those  given 
above)  and  a general  evaluation. 

Administrative  Methods 

In  order  to  assure  adequate  professionalism  and  required 
standardization  in  the  conduct  of  the  TCAC,  a formal  Administra- 
tor's Manual  (Wiesen  & Siegel,  1975)  was  developed.  The  manual 
which  was  116  pages  in  length  contained  details  relative  to.- 

a.  The  nature,  scope,  and  purpose  of  the  present 
program  and  background  information  on  assess- 
ment centers. 

b.  Detailed  administrative  instructions  for  each  ex- 
ercise including,  but  not  limited  to;  a precise, 
explicit  script  to  be  followed  in  the  learning 
phase  (where  applicable),  test  administrative 
instructions,  and  timing  and  scoring  instruc- 
tions. 

c.  Samples  of  the  various  forms  to  be  completed 
by  the  assessors  relative  to  various  exercises 
and  judgments. 

d.  The  schedule  to  be  followed  in  the  TCAC. 

c.  Privacy  righ'~  of  the  individuals  assessed. 

f.  Methods  of  managing  any  special  problems 
which  might  arise. 

In  addition  to  this  Manual,  formal  assessor  training  was  im- 
plemented. This  training  involved  3.  0 days  and  included:  (a)  a de- 
tailed, formal  review  and  elaboration  of  the  materials  in  the  manual, 
(b)  practice  exercise  administration  by  each  administrator  with 
critique  by  exercise  developers,  (c)  a dry  run  of  one  day's  opera- 
tion of  the  TCAC  assessing  Navy  enlisted  personnel  as  assessees 
and  employing  the  precise  methods  to  be  employed  in  the  conduct 
of  the  TCAC,  and  (d)  review  of  the  dry  run  by  total  TCAC  staff. 
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Exercises 


A summary  of  the  formal  exercises  included  in  the  TCAC 
is  given  in  the  subsequent  paragraphs.  Note  that  some  of  the  ex- 
ercises can  be  group  administered,  while  others  involve  individu- 
al administration. 


Conceptual  Integration/ Application  (Troubleshooting)  Exercise 


• Purpose 


The  Conceptuallntegration/ Application  evaluative  situation 
was  designed  to  measure  capability  to  integrate  simple  facts  and  re- 
lationships with  known  information  and  to  derive  appropriate  conclu- 
sions. To  perform  the  task,  a person  must  first  be  able  to  learn  and 
remember  certain  rules.  He  must  be  able  to  perceive  relationships 
and  he  must  be  able  to  manipulate  several  facts,  rules,  and  relation- 


ships ir  his  mind  at  the  same  time.  This  ability  is  often  called  "com- 
mon sense.  " Within  this  evaluative  situation,  the  operation/function/ 
logic  of  a simple  simulated  electromechanical-hydraulic  system  is 
taught.  Then  malfunctions  are  inserted  into  the  system.  The  task  of 
the  person  being  evaluated  is  to  state  the  cause  of  the  various  malfunc- 
tions as  thej'  are  presented.  Guilford's  Structure-of-lntellect  model 
suggests  that  the  following  fundamental  aspects  of  intellect  are  re- 
flected by  this  task:  cognition  and  memory  operations,  and  symbolic 
and  semantic  contents. 

• Outline  of  Procedure 

A pretest  instructional  session  provides  all  the  needed  infor- 
mation about  a hypothetical  system  including  the  function  of  each 
part,  the  logic  of  operation,  and  the  interrelationship  among  the 
parts.  Various  possible  malfunctions  and  their  probable  cause(s) 
are  also  presented  and  briefly  discussed.  The  test,  which  is  oral  in 
nature,  consists  of  presentation  of  various  system  malfunctions  for 
diagnosis  by  the  individual(s)  being  assessed.  The  most  likely  cause(s) 
for  each  malfunction  situation  is  indicated  by  the  tested  individual^, ' on 
a formalized  response  sheet. 

This  evaluative  exercise  may  be  administered  to  groups  of 
approximately  12  people.  A total  of  about  50  minutes  is  required  for 
the  explanatory  session  and  the  formal  test.  Figure  2 shows  this  ex- 
ercise being  administered  to  three  individuals. 
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• Scoring  Summary 


The  total  number  of  individual  and  multiple  correct  re- 
sponses is  determined. 

Coordinative  Speed  and  Accuracy  .Exercise 

• Purpose 

The  Coordinative  Speed  and  Accuracy  exercise  was  designed 
to  assess  ability  to;  organize  and  implement  a systematic  approach 
to  work,  pay  attention  to  details,  maintain  short  term  and  directed 
concentration  on  a task,  and  perceive  relationships  among  parts  to 
produce  a required  whole.  Completion  of  the  task  requires  working 
to  achieve  a goal  having  been  given  the  parts.  Accordingly,  the  situ- 
ation allows  for  observations  relative  to  an  individual's  approach  to 
work. 

This  evaluative  situation  also  involves  the  ability  to  use  small 
tools,  to  make  simple  measurements,  to  work  with  small  parts,  and 
to  perform  precise  work  relative  to  a set  of  standards.  These  abil- 
ities are  especially  relevant  to  those  Navy  ratings  in  which  independ- 
ent work  is  involved  and  to  ratings  which  require  the  use  of  small 
hand  tools  and  manipulation  of  small  parts. 

The  Guilford  Structure-of-Intellect  factors  subsumed  by  this 
test  are:  cognition  of  figural/ symbolic  units,  and  cognition  of  figural/ 
symbolic  relations. 

• Outline  of  Procedure 

An  instruction  and  practice  session  provides  the  individual 
with  an  opportunity  to  learn  to  perform  a simple  terminal  board  wir- 
ing task. 

In  the  test  situation,  the  individual  is  asked  to  put  this  new 
knowledge  to  use  by  connecting  terminals  on  one  terminal  strip  to 
terminals  on  a second  terminal  strip.  These  connections  are  made 
according  to  a wiring  diagram  and  a color-coding  chart  using  crimp- 
on  connectors  attached  to  7"  wire  lengths.  Figure  3 shows  the  tools 
and  equipment  supplied. 
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This  evaluative  situation  may  be  administered  to  a group 
of  up  to  four  people.  A total  of  about  35  minutes  is  required  for 
the  instruction,  practice,  and  test  periods.  The  test  is  adminis- 
tered as  a timed  test. 

• Scoring  Summary 

The  completed  work  is  scored  considering  adequacy  of  the 
following:  wire  length,  wire  stripping,  crimp-on  connections,  ter- 
minal screw  tightening,  choice  of  color  of  wire,  aueouacy  of  con- 
nections between  terminals,  and  procedure  employed  in  doing  the 
task. 

Separate  scores  are  derived  for  speed  (number  of  items 
completed),  accuracy  (percentage  correctly  completed),  and  a 
combination  of  speed  and  accuracy. 


Dual  Task  Exercise 
• Purpose 

The  Dual  Task  evaluative  situation  was  designed  to  measure 
ability  to  perform  in  a time-sharing  context.  This  type  of  task  is 
common  to  a number  of  Navy  watch  standing  situations.  Perform- 
ance of  two  unrelated  tasks  is  involved  in  the  Dual  Task  exercise. 

The  situation  requires  the  individual  to  keep  track  of  and  to  perform 
two  tasks  concurrently.  In  the  field  of  information  processing,  this 
is  called  time  sharing.  Time  sharing  is  a part  of  Navy  and  of  every- 
day life.  The  successful  and  joint  completion  of  the  tasks  involved  is 
dependent  on  this  ability. 

The  two  tasks  involve  simulated  watch  standing  on  a conti  ol- 
display  panel  and  fabricating  a pipe  assembly.  These  particular 
tasks  were  chosen  for  two  reasons.  First,  performance  of  tasks  re- 
quiring the  abilities  subsumed  by  these  tasks  is  likely  to  be  required 
of  many  E4  level  personnel.  Second,  the  Structure-of-Intellect  mod- 
el developed  by  Guilford  identifies  several  components  of  tiiese  tasks 
as  being  fundamental  aspects  of  intellective  performance.  Using 
Guilford's  terminology,  these  include  the  cognitive  and  evaluative 
operations  as  well  as  figural  and  symbolic  content. 
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• Outline  of  Procedure 

Simultaneously7,  and  with  a time  limit,  a pipe  assembly  it 
fabricated  and  a display-control  panel  is  monitored.  Several  times, 
and  in  response  to  an  indicator  on  the  panel,  the  individual  under 
assessment  is  required  to  make  adjustments  to  the  controls.  The 
required  control  adjustments  are  made  in  response  to  a randomly 
appearing  "warning.  " When  the  warning  signal  appears,  the  in- 
dividual responds  by  making  a set  of  precharted  responses  by  seri- 
ally "activating"  eight  control  unobs.  For  the  monitoring  task,  the 
amount  of  time  taken  to  respond  is  scored  along  with  the  accuracy 
of  control  adjustments.  The  concomitantly  performed  pipe  assem- 
bly task  requires  the  individual  being  assessed  to;  (a)  read  a sche- 
matic, (b)  measure  lengths  of  pipe,  (c)  select  the  type  of  fittings 
and  the  diameter  pipes  and  fittings  indicated  on  the  schematic,  and 
(d)  perform  the  actual  assembly'.  Figure  4 presents  the  exercise 
equipment.  The  vertical  panel  shown  in  Figure  4 contains  the  con- 
trol-display assembly7,  while  the  horizontal  panel  contains  the  nec- 
essary components  for  the  pipe  assembly  task.  A number  of  "dis- 
tractor"  (wrong)  components  is  also  included. 

This  evaluation  is  administered  individually7  with  a 12  min- 
ute time  limit. 

The  pretest  training  situation  for  this  exercise  consists  of 
a formal  training  period  which  involves  instructions  in  both  how  to 
set  the  control  panel  when  a warning  signal  occurs  and  how  to  read 
a schematic  and  make  a pipe  assembly.  Hands  on,  one-to-one 
training  and  practice  in  pipe  assembly  is  also  given. 

• Scoring  Summary 

For  the  monitoring  task,  the  number  of  correct  control  set- 
tings and  the  integrated  time  to  respond  are  scored.  For  the  pipe 
assembly  task,  the  number  of  correctly  selected  parts  and  the  num- 
ber of  complete  and  correct  connections  are  scored. 
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Inspectional/Sort  Exercise 
Purpose 

The  abiLity  to  make  reasonable  discriminations  among  mean- 
ingful stimuli  is  basic  to  all  forms  of  learning  and  behavior.  The 
Inspection/ Sort  evaluation  assesses  the  ability  to  discriminate 
among  objects  that  are  similar  to  many  found  in  Navy  jobs.  In  ad- 
dition to  the  ability  to  evaluate  similarity,  the  ability  to  notice  im- 
perfections is  evaluated. 

This  test  requires  the  use  of  the  cognition,  memory,  and 
evaluation  operations  coupled  with  figural  content  in  the  Guilford 
scheme. 

Outline  of  Procedure 


Ninety  objects,  15  each  of  6 types,  are  sorted  by  the  indi- 
vidual into  types  (see  Figure  5).  A time  limit  is  imposed.  Some 
of  the  objects  have  imperfections  or  do  not  closely  match  the  others. 
The  individual  being  assessed  places  these  into  a reject  bin.  A 
"standard"  comparison  object  of  each  type  is  displayed  to  the  per- 
son being  assessed  at  all  times. 

This  evaluation  may  be  administered  to  groups  of  any  num- 
ber of  persons.  The  evaluation  has  a two-minute  time  limit. 

Scoring  Summary 
Four  scores  are  derived: 

1.  A speed  score- -number  of  items  sorted/ rejected. 

2.  An  accuracy  score  for  good  items--percentage 
correctly  sorted  items  (based  on  total  items 
sorted ) . 

3.  An  accuracy  score  for  imperfect  and  mis- 
matched items--percentage  of  correctly  re- 
jected items  (based  on  items  sorted). 

4.  A speed  and  accuracy  score  — number  of  items 
correctly  sorted  plus  number  of  items  correct- 
ly rejected. 
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The  Level  of  Aspiration  evaluative  situation  was  designed 
to  assess,  among  other  things,  need  for  achievement  (the  level  of 
the  goals  that  a person  sets  for  himself).  This,  in  turn,  is  re- 
flected by  general  motivation  and  specific  goal  striving.  Level  of 
aspiration  is  generally  more  realistic  in  the  more  secure  person. 

The  reality  orientation  of  the  level  of  aspiration  along  with  reaction 
to  success  and  failure  (e.  g.  , tendency  to  give  up  easily)  are  also 
evaluated.  The  task  also  provides  an  opportunity  for  some  informal 
conversation  between  the  individual  being  assessed  and  the  adminis- 
trator. When  this  occurs,  the  ability  of  the  individual  to  communi- 
cate, as  well  as  his  general  demeanor,  can  be  judged. 

• Outline  of  Procedure 

Behavior  during  a standardized  dart  throwing  game  provides 
the  situational  basis  for  the  Level  of  Aspiration  evaluation.  A spe- 
cific game  sequence  is  followed  which  yields  estimated  and  actual 
performance  on  the  task.  Specifically,  the  individual  being  assessed 
is  asked:  to  make  an  initial  performance  estimate  in  view  of  a stated 
group  performance  average,  to  throw  three  darts  and  obtain  an  initial 
score,  to  make  a second  performance  estimate  in  view  of  his  actual 
performance,  his  original  estimate,  and  the  group  average,  and  to 
throw  three  more  darts;  this  procedure  is  repeated  fi">r  three  dart 
throwing  trials.  The  group  average,  the  estimates,  and  the  attained 
scores  are  written  on  a blackboard  which  is  visible  to  the  individual 
being  assessed,  as  the  scores  are  obtained.  These  are  permanently 
recorded  at  the  conclusion  of  the  test.  Evaluation  of  behavioral  traits 
is  made  at  the  end  of  the  test  using  prepared  scales  and  free  form  de- 
scription (when  appropriate). 

This  evaluation  is  individually  administered  and  requires  ap- 
proximately 10  minutes. 

• Scoring  Summary 

Each  estimate  and  attained  score  is  copied  from  the  blackboai  d 
to  a data  sheet  at  the  end  of  the  evaluation.  The  initial  estimates  and 
the  attained  scores  y'eld  estimates  of  realism.  Comparison  of  the  giv- 
en group  averagt  with  the  estimates  of  the  individual  being  assessed 
yields  a measure  of  need  for  achievement.  Similarly,  comparison  of 
the  attained  and  the  predicted  scores  yields  an  estimate  of  realism. 
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Reliability  Exercise 


• Purpose 

Individual  morality  and  integrity  are  basic  to  the  success- 
ful operation  of  our  society  including,  of  course,  the  Navy.  Among 
psychologists,  there  is  debate  as  to  whether  or  not  verbal  self  re- 
ports relative  to  morality  will  be  reflected  in  actual  behavior.  In 
the  present  evaluative  situation,  this  area  of  debate  is  bypassed  by 
looking  at  a behavioral  sample  ot  actual  moral  behavior.  The  Re- 
liability evaluative  task  is  designed  to  provide  a situation  during 
which  cheating  is  seemingly  possible,  undetectable,  and  beneficial. 
The  individual  is  left  alone  during  this  evaluation.  Accordingly.  Hu- 
moral decision  to  cheat  or  not  to  cheat  is  solely  his,  independent  of 
the  behavior,  encouragement,  or  discouragement  of  others.  Thus, 
this  evaluative  situation  reflects  intrinsic  rather  than  extrinsic  n >ral 
motivation. 

• Outline  of  Procedure 

In  this  evaluative  situation  (Figure  (j).  the  individual  - ;>  -m-iI 
to  perform  an  easy  task  (needle  threading)  which  requires  little  in- 
struction. to  score  his  own  performance,  and  to  reset  the  apparatus 
to  its  original  condition.  While  he  is  performing  the  task,  the  ad- 
ministrator attends  to  a visible  and  ostensibly  pressing  task.  The 
evaluative  task  is  graded  in  difficulty  as  a result  of  using  a set  of 
needles  with  graded  eye  size.  The  10  needles  w ith  the  largest  eye 
size  are  rather  easy  to  thread.  The  five  needles  with  the  smallest 
eyes  are  impossible  to  threat’  because  their  eyes  are  blocked  with 
a transparent  plastic.  Therefore,  the  actual  maximum  score  pos- 
sible is  less  than  the  apparent  maximum.  If  greater  than  10.  the 
reported  score  is  fallacious. 

This  evaluative  situation  is  individually  administered  with  a 
2.  5 minute  time  limit.  Figure  6 sho.vs  one  person  taking  this  exer- 
cise. 


The  individual's  self  reported  score  is  considered  truthful 
if  it  is  less  than  or  equal  to  10.  A fallacious  score  can  range  from 
11  to  15  depending  upon  how  much  the  individual  being  assessed  in- 
flates his  performance  report  over  the  attainable  maximum  of  10. 


Social  Interactive  Exercise 


• Purpose 

When  working  in  a group,  some  people  make  contributions 
that  are  more  important  and  facilitating  to  the  group  goals  than 
those  made  by  others.  Their  ideas  and  suggestions  are  more  likely 
to  be  accepted  by  the  group  than  those  made  by  other  group  mem- 
bers. People  who  have  ideas  and  suggestions  which  are  accepted 
by  others  are  said  to  have  leadership  abilities. 

The  Social  Interactive  evaluation  situation  provides  the  as- 
sessor an  opportunity  to  observe  various  individuals  being  assessed 
interacting  in  a standardized  group  task  setting  and  to  note  the  pres- 
ence of  leadership  and  positive  social  interactive  skills  in  the  indi- 
viduals under  assessment.  The  task  is  designed  to  provide  an  oppor- 
tunity to  see  how  well  each  individual  cooperates  and  works  within 
the  group  while  the  group  is  under  go..1  achievement  pressure. 

The  exercise  situation  is  designed  to  involve  increasing  out- 
side pressure  for  cooperation  and  leadership  over  trials.  These  be- 
haviors are  monitored  by  the  administrator(s)  and  rated  during  each 
phase  of  the  assigned  group  task. 

• Outline  of  Procedure 

The  individuals  under  assessment  are  asked  to  perform  a 
simulated  resupply  task,  once  with  little  urging  and  no  time  to  con- 
template cooperation,  and  twice  with  the  strong  suggestion  and  the 
time  to  plan  a closely  coordinated  team  effort.  Data  reflecting  co- 
operation and  leadership  evidenced  by  each  individual  during  each 
stage  are  recorded  using  a standard  behavioral  scoring  system. 

This  exercise  is  administered  to  groups  of  three  to  six.  The 
task  requires  five  consecutive,  three-minute  periods. 

• Scoring  Summary 

Observational  data  are  recorded  for  each  individual  for 
each  phase  of  the  evaluation.  Attention  is  given  to  behavioral  in- 
dications of;  leadership,  cooperation,  deportment,  enthusiasm, 
ability  to  communicate  clearly,  attention  to  task,  rule  following, 
argumentativeness,  and  number  and  quality  of  ideas.  Integrated 
scores  and  summaries  of  other  comments  are  derived. 
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Tool  and  Object  Nomenuature,  Use,  and  Recognition  Exercise 


• Purpose 

The  Tool  and  Object  Nomenclature,  Use,  and  Recognition 
exercise  is  designed  to  assess  several  aspects  of  the  ability  to  form 
and  remember  simple  associations.  The  specific  task  chosen  to  al- 
low these  evaluations  asks  the  individual  to  learn  to  associate  the 
names  of  several  objects  taken  from  Navy  life  with  the  actual  objects 
and  their  uses.  Three  recall  situational  aspects  a -e  measured:  (a) 
recall  of  use  based  on  observation,  (b)  recall  of  names  based  on  ob- 
servation. and  (c)  recall  of  names  based  on  verbal  description. 

These  types  of  abilities  are  required  to  some  extent  in  all  Navy 
jobs.  Guilford's  Structure-of-Intellect  model  identifies  several 
aspects  of  this  evaluative  task  as  fundamental  to  intellective  func- 
tion. Using  Guilford's  terminology,  these  include  the  operations: 
cognition  and  memory,  and  the  contents:  figural,  symbolic,  and 
semantic. 

• Outline  of  Procedure 


In  a pretest  learning  situation,  typical  tools  and  objects  taken 
from  Navy  life  are  presented.  Their  names  and  usage  are  described 
and  briefly  discussed.  This  presentation  is  followed  by  a verbally 
administered  true-false  test.  Three  scores  are  derived  which  re- 
flect learning  and  memory  of  the  information  given.  This  eval  1a- 
tion  may  be  administered  to  groups  of  up  to  about  12.  A total  time 
of  about  one  hour  is  required  to  administer  the  two  parts  of  this 
evaluation.  Figure  7 presents  the  administration  of  a part  of  this 
exercise. 

• Scoring  Summary 

Each  of  the  three  different  types  of  questions  is  scored  us- 
ing its  own  key.  The  number  of  correct  answers  minus  one-half 
the  number  of  incorrect  answers  is  calculated  for  each  subtest.  A 
grand  total  is  also  calculated. 
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Writing  Sample  Exercise  (Optional) 


• Purpose 

Some  ratings  require  a high  level  of  writing  ability,  while 
others  require  only  minimal  writing  skill.  Only  those  individuals 
who  express  an  interest  in  a rating  which  requires  moderate  to 
highly  developed  writing  skills  were  asked  to  complete  the  Writing 
Sample  exercise. 

In  the  W'riting  Sample  situation,  the  individual  is  given  the 
opportunity  to  write  a short  (up  to  300  words)  response  to  two  ques- 
tions with  a 30  minute  time  limit  for  both  questions.  One  question 
is  designed  so  that  all  individuals  will  have  about  equal  knowledge 
on  the  subject.  This  question  concerns  boot  camp  activity.  The 
second  is  designed  to  allow  the  individual  to  make  some  evaluative 
judgments  concerning  his  present  Navy  assignment.  Accordingly, 
in  addition  to  the  sample  of  writing  beha.vior  (which  will  provide 
measures  of  writing  skill),  the  sample  reflects  some  of  the  values 
and  goals  of  the  individual. 

• Outline  of  Procedure 

The  individual  under  assessment  is  read  and  given  a written 
copy  of  two  questions.  The  individual  has  30  minutes  to  write  re- 
sponses to  these  questions. 

This  evaluation  may  be  administered  individually  or  to 

groups. 


• Scoring  Summary 

For  each  question,  a score  is  obtained  for;  number  of  cor- 
rectly spelled  words,  percentage  of  words  correctly  spelled,  num- 
ber of  complete  sentences,  percentage  of  complete  sentences,  in- 
telligibility, and  insight. 
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Semistructured  Interview 


• Purpose 

The  purpose  of  the  semistructured  interview  is  to  provide  j 

evaluative  and  predictive  information  not  provided  elsewhere  in 
the  information  substrate  for  an  assessee.  The  interviewer  has 
available  for  his  use  all  evaluative  information  generated  up  to 
th ■■  time  of  the  interview  of  a given  assessee.  Additionally,  the 
folder  for  a given  assessee  contains  background  data  as  available 
from  the  "service  jacket"  of  the  individual  and  related  "basic  bat- 
tery" test  results.  The  end  result  of  the  interview  is  a synthesis 
of  these  materials  into  a meaningful  nexus  with  additional  informa- 
tion provided  which  will  fill  in  gaps  relative  to  the  purposes  of  the 
Center- -providing,  if  possible,  a "career  field"  recommendation 
for  a given  assessee. 

The  interviewer  is  a person  who  is  trained  in  and  skilled  at 
interview  methods.  Additionally,  he  has  experience  and  knowledge 
in  personality  dynamics,  motivational  theory,  and  the  like. 

• Content  of  Interview 

Because  it  is  important  that  all  interviewers  use  somewhat 
the  same  approach  and  questions,  a semistructured  format  is  involv- 
ed. This  helps  to  assure  between  interviewer  reliability  in  content. 

However,  any  area  of  import  vis-a-vis  a specific  assessee  is  pur- 
sued in  sufficient  depth  to  satisfy  the  interview  purposes.  The  three 
general  areas  which  are  probed  in  each  interview  are:  interests/ 
background,  motivation,  and  general  capacity. 

• Interview  Report 

An  interview  checklist  rating  form  is  completed.  Each  of  16 
items  is  rated  on  a 4 point  scale.  A checkoff  is  provided  for  items 
that  cannot  be  rated  based  on  the  interview.  Space  is  provided  at  the 
end  of  the  checklist  form  to  enter  any  justifications  (examples)  for 
the  ratings  and,  more  importantly,  for  entering  any  additional  in- 
formation which  may  be  important  for  the  use  of  the  panel  when  it 
considers  the  individual  under  evaluation. 
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Briefing/Debriefing  and  TCAC  Evaluation  Form 


i i 


Two  group  meetings,  one  at  the  beginning  and  one  at  the 
end  of  each  day  on  which  the  TCAC  operated,  were  held  to  answer 
any  questions  of  the  individuals  under  assessment  and  to  review 
the  nature,  scope,  and  purpose  of  the  program. 


At  the  end  of  the  day's  meeting  (Figure  8)  each  group  of 
individuals  under  assessment  was  asked  to  provide  verbal  and 
written  feedback  about  the  several  aspects  of  the  Center's  opera- 
tions, including:  perceived  fairness  of  the  tests  and  relative  pref- 
erence for  the  various  tests.  A standard  response  form  was  devel- 
oped for  this  purpose.  The  morning  meeting  included  a discussion 
and  a verbatim  reading  of  a statement  concerning  rights  to  freedom 
of  information  and  privacy. 

Administrative  Procedures  for  TCAC 


Wiesen  and  Siegel  (1975)  described,  in  detail,  all  aspects 
of  the  operation  of  the  TCAC,  including,  but  not  limited  to,  prepar- 
ing for,  administering  and  scoring  the  exercises,  scheduling,  and 
TCAC  decision  procedures.  Briefly  stated,  the  daily  procedure  in- 
volved: briefing  the  group  of  individuals  to  be  assessed  that  day  rela- 
tive to  the  purpose,  content,  and  goals  of  the  TCAC  (15  min.  ),  ad- 
ministration of  the  set  of  exercises  (5  hours),  and  debriefing  (15  min.). 


Weekly  meetings  of  the  Center's  staff  were  held  during  which 
each  person  under  assessment  was  individually  considered  and  in 
which  a classification  and  a set  of  ratings  were  completed.  The 
procedure  for  this  weekly  meeting  was  prescribed  in  advance  in 
the  administrator's  manual  (Wiesen  & Siegel,  1975).  To  facilitate 
these  weekly  discussions,  there  was  a meeting  of  the  Center's  as- 
sessors at  the  end  of  each  day.  In  this  meeting,  each  individual 
tested  that  day  was  assigned  to  one  administrator  who  would  serve 
as  an  advocate  for  that  individual  at  the  subsequent  weekly  meeting. 
The  advoca+e  for  an  individual  summarized  the  relevant  data,  gave 
his  evaluat  on  of  the  individual  on  the  three  dimensions  discussed 
earlier,  and  made  an  initial  proposal  of  career  field  for  that  per- 
son. The  discussion  which  followed  always  began  with  a statement 
from  the  assessor  who  interviewed  the  individual  being  assessed. 
Then,  other  contributions  of  information  or  observations  concern- 
ing the  individual  were  made  by  the  other  assessors.  A discus- 
sion of  the  merits  proposed  for  the  individual  then  ensued.  This 
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was  followed  by  a secret  ballot.  Each  administrator  voted  eith- 
er to  accept  or  to  reject  the  proposal  and  to  make  a prediction 
of  the  anticipated  level  of  on-the-  job  success  in  that  rating.  In 

I the  event  of  a defeated  proposal,  one  of  two  avenues  could  be  fol- 

lowed; (a)  a second  proposal  could  be  made,  or  (I))  if  there  was  no 
second  proposal,  the  individual  was  recommended  for  general  rates. 
In  the  event  of  a tie  vote,  a second  discussion  period  and  vote  oc- 
curred. If  there  was  still  a tie  vote,  then  the  Center's  director 
cast  a "tiebreaker"  ballot. 

Individuals  Assessed 


Enlisted  personnel  (N  - 140)  in  the  Fireman  or  Seaman  Ap- 
prentice Training  Schools  at  the  Naval  Training  Center,  San  Diego, 
California,  were  assessed  by  the  TCAC.  Within  limits  of  a prac- 
tical nature,  the  individuals  were  chosen  to  be  not  A school  quali- 
fied four  year  obligators,  with  basic  battery  GCT  and  ARI  scores 
totaling  less  than  100. 

Description  of  Sample 

Mean  sample  scores  on  the  General  Classification  Test  (GCT), 
Arithmetic  (ARI),  Mechanical  (MECH),  Clerical  (CLER),  Electronic 
(ETST),  and  Shop  (SHOP  PR)  tests  are  given  in  Table  4.  These  tests 
were  administered  by  the  Navy  to  all  enlisted  personnel  on  entry  into 
the  Navy  and  are  designed  to  have  a mean  of  50  and  a standard  devi- 
ation of  10.  The  means  for  the  present  sample  are  less  than  50  for 
all  but  the  SHOP  PR  test.  Four  of  these  means  (GCT,  ARI.  MECH, 
and  ETST)  are  significantly  below  50.  The  group  standard  devia- 
tions are  less  than  10  for  all  six  tests.  These  differences  from  tot- 
al norms  were  to  be  anticipated.  The  population  of  enlisted  men  in 
the  Apprentice  School  is  generally  restricted  in  range  to  the  lower 
half  of  the  distributions  of  these  "basic  battery"  test  scores.  The 
higher  scoring  men  typically  are  assigned  to  an  A school.  Thus, 
the  desired  sample  of  low  aptitude  enlisted  personnel  was  assessed. 

The  group  received  a mean  of  2.  1 demerits  in  basic  train- 
ing, This  demerit  level  is  not  considered  to  be  unusual. 

The  mean  age,  level  of  schooling,  and  reported  arrest 
record  were,  respectively,  19,  12,  and  21.  Accordingly,  the 
sample  contained,  on  the  average,  recent  high  school  graduates, 
about  21  per  cent  of  whom  reported  an  arrest  record. 
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Mean  and  Standard  Deviation  of  Sample  on  Navy  Related  Scores  and  Biographical  Item 


TC’AC  Staff  and  Time  of  Operation 


The  Director  of  the  TCAC  and  one  other  administrator  were 
psychologists.  The  remaining  two  administrators  were  Navy  Chief 
Petty  Officers.  It  was  believed  that  this  administrator  mix  would 
provide  a balanced  staff  of  persons  who  represented  both  the  Navy 
job  requirements  and  the  personality  and  individual  traits  points 
of  view.  The  Director's  position  was  filled  by  two  people  during 
the  course  of  operation  of  the  Center  to  make  a total  of  five  ad- 
ministrators who  were  directly  associated  with  the  day-to-day 
operations.  The  TCAC  was  in  operation  for  approximately  seven 
weeks  during  the  months  of  November  and  December,  1975. 

Data  Considered 

The  data  considered  for  each  individual  assessed  and  clas- 
sified included  2 8 scores  resulting  from  the  Center's  exercises, 
biographical  data,  observations,  7 scores  taken  from  Navy  re- 
cords, and  10  scores  reflecting  the  Center's  overall  evaluation. 

The  substance  of  these  data  is  summarized  in  Table  5. 
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Table  5 


Number  and  Source  of  Data  Scores  Recorded 
for  Each  Individual  Assessed/ Classified 


Number  of 

Source  Scot e s 

lenter  Exercise 

Center  Exercise  3 


Center  Exercise 


Center  Exercise 


u 


Center  Exercise 


7 


Center  Exercise  1 


Center  Exercise  3 


Center  Exercise  u 


Center  Exercise 


Center  Observation  2 

Center  Observation 
Center  Observation 

Navy  Records 

Center  Evaluation  1 

1 
4 

1 

3 


Source  of  Scores 


Conceptual  Integration/ Application 

number  of  co erect  troubleshooting  items 


Coer dinative  Speed  and  Accuracy 
speed 
accuracy 

speed  and  accuracy 


Dual  Task 

pipe  assembly  (total) 

control  watch  (numoer  of  correct  settings 
and  latency) 

Inspection/Sort 
speed  score 

accuracy  score  for  good  times 

accuracy  score  for  imperfect /mismatched  items 

speed  and  accuracy  score 

Level  of  Aspiration 

first  estimate  by  individual 
sum  of  remaining  estimates 
sum.  of  realism  scores 
sum  of  pessimism  scores 
rea-ism  evaluation  (dichotomous ' 
optimism  evaluation  (dichotomous) 
striving  evaluation  (dichotomous) 

Reliability 

lie-no  lie  dichotomy 

Social  Interactive 
phase  1-3  total 
phase  4-9  total 
total 

Tool  ana  Object  Nomenclature,  Use, and  Recognition 
observation  recall — use 
observation  recall — name 
verbal  recall — name 
total 

Writing  Sample 

This  exercise  was  optional.  Only  13  assessees 
elected  this  exercise.  It  is  not  considered 
further. 

Interview 

weighted  sum  of  evaluations 
number  of  items  not  judged 

Incidental  Observations 

Biographical  Data 

educational  background 
arrest  record 

Data  from  Navy  Records 

test  scores  (basic  battery) 
basic  training  disciplinary  record 

career  field  proposed  (yes-no  dichotomy) 

career  field  proposed  (spec -fie  field) 

forecast  of  on-the-job  perform*’  ce  level:  4 point 
scale  (recorded  separately  for-  each  administrator) 

number  of  proposals  considered  for  an  individual 

evaluation  of  each  assessee  on  each  of  three 
dimensions:  learning,  psvehomotor,  and  social 
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RESULTS  AND  INTERPRETATION 


Career  Field  Recommendations 

The  TCAC  arrived  at  two  decisions  relative  to  each  man 
processed:  a career  field  recommendation,  and  a prediction  of 
job  success  in  that  career  field  after  six  months  on  the  job. 

Career  field  recommendations  were  based  on  the  evalua- 
tions and  on  the  career  field  requirements  as  determined  by  the 
job  analysis  described  in  the  introductory  chapter. 

The  career  field  recommendations  of  the  TCAC  staff  were 
quite  diversified  given  the  background  of  the  men  in  the  sample. 

All  of  the  enlisted  men  assessed  had  been  previously  selected  for 
general  duty  on  completion  of  the  Apprentice  School.  These  men 
were  considered  on  the  basis  of  the  usual  classification  procedure 
to  be  generally  unqualified  for  immediate  entry  to  a specific  rat- 
ing through  A school  training.  Of  the  total  sample  assessed,  79 
per  cent  had  achieved  a combined  GCT  and  ARI  score  of  less  than 
100.  Despite  this,  the  TCAC  recommended  54  per  cent  of  the  men 
assessed  for  career  fields  (specific  ratings).  This  suggests  some 
sensitivity  to  the  TCAC  process  not  found  in  the  usual  Navy  classi- 
fication process.  The  remaining  46  per  cent  were  recommended 
to  continue  in  the  general  rates.  The  specialty  most  often  (27  per 
cent)  recommended  was  Machinist's  Mate.  The  complete  distribu- 
tion of  recommendations  is  given  in  Table  6. 

Predictions  of  On-the-Job  Success 

Each  administrator  made  a prediction  of  on-the-job  suc- 
cess for  each  individual  assessed  relative  to  the  assigned  career 
field.  The  administrators  made  this  rating  on  the  basis  of  their 
interpretation  of  the  data  available  for  each  individual  assessed 
and  classified.  These  predictions,  made  on  a scale  ranging  from 
0 to  3,  were  summed  to  get  an  overall  level  of  on-the-job  success 
prediction  for  each  assessee.  This  combined  score  could  range 
from  0 to  12. 


i 
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Table  6 

Career  Field  Recommendations  and  Corresponding 
Mean  Predictions  of  Job  Success 


Number 


Percen t 


Mean  Predicted  Level 


Career  Field 

Recommended 

Recommended 

of  on  the  j 

General  Rates 

64 

45.7 

8.6 

Postal  Leri  [PC ) 

11 

7.9 

10.  7 

Storekeeper  (SK) 

14 

10.  0 

10.9 

Yeoman  (YN) 

6 

4.3 

in 

00 

Signalman  (SM) 

2 

1.4 

9.0 

M i hinl  t's  Ma te  { MM ) 

38 

27.1 

CO 

CD 

Hosp ital 

Corp sman  (HM) 

O 

2.1 

9.  6 

thei  Pic 

Rating 

3 

1.4 

11.7 

TOTAL 


140 


weighted  mean  9.4 


*s.j  res  range  from  0 to  12 


The  mean  level  of  success  for  those  men  recommended  for 
general  rates  and  for  the  men  recommended  for  specific  ratings 
are,  respectively,  3.  6 and  10.  1.  Details  of  these  predictions  are 
given  in  Table  6.  The  mean  prediction  of  success  for  the  men 
recommended  for  general  rates  was  neither  the  lowest  nor  the 
highest  of  the  mean  predictions  for  the  various  career  fields.  It 
was,  however,  lower  than  the  means  of  most  of  the  groups  of 
men  recommended  for  a specific  career  field.  This  lack  of  a 
simple  relationsh'p  between  the  level  of  difficulty  of  the  job  and 
the  predicted  level  of  on-the-job  performance  appears  to  be  due 
to  joint  consideration  of  the  man  and  the  career  field  recommen- 
dation made  for  that  man.  The  specific  career  field  was  chosen 
based  on  the  man's  global  characteristics.  The  prediction  of  lev- 
el of  job  success  was  then  made  for  the  man  in  the  rating  suggest- 
ed. For  example,  a man  who  was  evaluated  highly  by  the  TCAC 
and  therefore  recommended  for  a rating  (e.  g.  , HM)  might  be  pre- 
dicted to  show  an  average  level  of  on-the-job  performance  due  to 
the  demanding  nature  of  the  rating  recommended.  Due  to  this,  the 
subsequent  analysis  of  predicted  level  of  job  performance  includes 
consideration  of  men  within  ratings  rather  than  simply  considering 
together  all  men  in  all  ratings. 
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The  mean  and  standard  deviation  of  each  set  of  TCAC  ex- 
ercise scores/judgments  was  calculated.  A description  of  these 
scores,  along  with  the  obtained  means  and  standard  deviations,  is 


presented  in  Table  7.  We  note  that  there  are  no  indications  of  the 
means  being  affected  by  either  a ceiling  or  a floor  effect.  More- 
over, in  each  case,  the  standard  deviation  seems  reasonable. 

Quite  obviously,  there  is  no  normative  group  with  which  these  dis- 
tributions may  be  compared. 


The  product  moment  correlations  along  all  scores  were  al- 
so calculated.  These  intercorrelations  were  generally  moderate 
but  not  high.  This  indicates  some  uniqueness  for  the  various  meas- 
ures, as  well  as  a degree  of  overlap.  Subscores  within  some  tests 
seem  to  be  highly  intercorrelated.  Accordingly,  a subset  of  16 
scores  was  chosen  to  be  used  in  most  of  the  further  analyses.  These 
16  scores  are  marked  with  an  asterisk  in  Table  7. 

The  intercorrelational  matrix  that  includes  these  16  TCAC 
measures,  as  well  as  the  Navy  related  scores,  biographical  items, 
and  the  several  TCAC  judgments  is  presented  in  Table  8. 


Dimensional  Judgments 


Each  person  processed  was  rated  on  each  of  the  three  dimen- 
sions-- learning,  psychophysical  and  motor,  and  social/motivational-- 
employed  as  a basis  for  the  job  analysis.  The  judgments  relative  to 
each  individual  assessed  were  made  by  only  one  administrator.  The 
mean  across  assesses,  for  each  of  the  three  dimensions,  was,  re- 
spectively, 3.  18,  3.23,  and  3.  19.  These  means  are  close  to  the 
midpoint  (3.  00)  of  the  1 to  5 rating  scale  used  and  are  quite  close 
to  each  other.  This  suggests  that  there  was  no  overall  bias  on  the 
part  of  the  administrators  to  rate  one  dimension  more  highly  than 
another. 


: 


NOTE : Dec  imd  1 points  omi 1 1 M . 


Inter  rater  Kel  [ability 
' 

The  interrater  reliability  of  the  administrators  was  inves- 
tigated. Four  administrators  each  scored  the  worx  of  from  5 to 
19  assessees.  An  intraclass  correlation  coefficient  (the  average 
of  the  correlation  coefficients  oetween  pairs  of  administrators) 
was  used  to  index  the  amount  of  agreement,  by  exercise,  between 
the  scores  reported  by  each  administrator  for  each  individual  as- 
sessed (Guilford,  19(15,  pp.  299-500). 

The  level  of  reliability  could  not  be  investigated  in  this  way 
for  all  measures.  Some  of  the  measures  for  some  of  the  exercises 
required  dismantling  or  otherwise  destroying  the  work  of  the  in- 
dividual being  assessed.  The  approach  to  reliability  used  with 
these  measures  was  to  derive  scores  for  those  aspects  of  the  exer- 
cise that  were  scorable  without  dismantling  or  destroying  the  prod- 
uct to  be  scored. 


The  intraclass  correlation  coefficients  indexing  interrater 
reliability  ranged  from  . 72  to  1.  00  with  a median  of  . 97  and  an 
interquartile  range  from  . 92  to  . 99.  These  reliability  coefficients 
are  presented  in  Table  9.  The  one  aberrantly  low  reliability  coef- 
ficient of  . 72  (accuracy  score  for  dual  task  exercise)  was  based  on 
the  fewest  number  of  subjects  (five).  In  fact,  one  administrator's 
relatively  large  discrepancy  regarding  the  score  of  one  assessee 
caused  this  lowered  reliability  coefficient.  All  remaining  19 
scores  involved  in  this  reliability  coefficient  were  unanimously 
agreed  on  by  the  four  administrators. 

I 

These  levels  of  reliability  compare  quite  favorably  with 
those  found  in  the  literature.  The  scores  were  all  of  tangible 
end  products.  Working  with  tangible  end  products,  Bornstein, 
Jensen,  and  Dunn  (1954)  reported  a mean  phi  coefficient  reflect- 
ing inter  rater  reliability  of  . 78  (with  a standard  deviation  of  . 19). 
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Table  9 

Interrater  Reliability  (Intraclass  Correlation) 
of  Selected  Center  Exercise  Scores 


Measure 

Number  of 
Subjects 
Represented 

Number  of 
Administra tors 
Represented 

Reliability 

Coefficient 

Coordinative  Speed  and 
Accuracy:  combined  and 
accuracy  score* 

19 

4 

.96 

speed  score* 

16 

4 

. 97 

accuracy  score* 

16 

4 

cn 

CO 

Dual  Task 

construction  score  6 4 .99 

accuracy  of  settings  5 4 .72 


Inspection/Sort : 


speed  score 

15 

4 

1.00 

accuracy  score 

15 

4 

.93 

accuracy  score 
combined  speed  and 

15 

4 

.92 

accuracy  score 

15 

4 

.99 

* Certain  measures  in  this  test  require  dismantling  the  assessee 's  work.  Only 
those  measures  that  are  nondestructive  were  scored  for  this  reliability 
estimate. 
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Dimensional  Evaluations 


The  three  dimensional  ratings  were  further  considered  in 
three  waj  tl)  inter  and  intradimensional  relationships  of  the 
measures,  (2)  consistency  of  the  policies  of  the  individual  admin- 
istrators, and  (3)  degree  of  agreement  among  the  administrators. 
These  two  types  of  accord  (intra-administrator  consistency  and  in- 
teradministrator agreement)  reflect  different  measurement  con- 
structs. Intra-administrator  consistency  is  a measure  of  the  con- 
sistency of  any  one  administrator  in  applying  his  decision  making 
polic v . The  interadministrator  agreement  is  a measure  of  how 
closely  the  policies  of  the  individual  administrators  agree.  This 
administrator  agreement  may  also  be  taken  as  an  indication  of  the 
interadministrator  reliability  relative  to  these  ratings.  A policy 
capturing  approach  to  decision  making  was  employed  in  the  analy- 
sis of  these  two  types  of  accord. 

Interdimensional  Relationships 

Three  Pearson  product  moment  correlation  coefficients 
were  calculated  considering  all  possible  pairs  of  dimensions, 
these  intercorrelations  are  given  in  Table  10.  The  common  vari- 
ability in  these  dimensions,  as  indexed  by  r2.  ranges  from  6 per 
cent  to  18  per  cent.  The  highest  relationship  was  between  the 
learning  and  the  psychophysical  and  motor  dimensions.  The  di- 
mensional measurements  seem,  accordingly,  to  have  been  inde- 
pendently considered  by  the  administrators. 


Table  10 

Intercorrelations  Among  Dimensional  Itatings 


Learn ing 

ychophy  ' il 
in  ; hy  Leal 


nearnmg 


Psychophysical 
and  Motor 
742 


Soc ial/ 
Motivational 
.27 


.25 
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Intradimensional  Relationships 


A 


The  various  TCAC  exercises  were  intended  to  measure 
abilities  along  three  dimensions--learning,  psychophysical  and  mo- 
tor, and  social/ motivational.  The  exercises  designed  for  the  TCAC 
were  mainly  complex  job  sample  learning  and  performance  tests. 
They  were  not  designed  to  measure  singular  or  simple  abilities.  As 
indicated  in  Table  3,  most  of  the  exercises  were  designed  to  span 
Guilford  Structure-of-Intellect  categories.  Accordingly,  some  in- 
tercorrelations between  any  two  exercises  might  be  expected  with 
the  highest  intercorrelations  occurring  among  exercises  reflecting 
any  one  dimension.  This,  indeed,  was  the  case. 

A comparison  was  completed  of  the  proportion  of  statistical- 
ly significant  product  moment  correlations  among  measures  of  one 
dimension  with  the  number  of  statistically  significant  correlation 
coefficients  between  two  dimensions.  The  critical  value  with  140 
assesses  for  p = . 05,  two  tailed  is  r = . 17.  Of  the  total  of  378  pos- 
sible comparisons  (28  scores  taken  two  at  a time),  251  are  inter- 
dimensional  and  12  7 are  intradimensional  correlations.  Of  the  inter- 
dimensional  correlation  coefficients,  20.  3 per  cent  (51)  were  statis- 
tically significant.  Of  the  intradimensional  correlations,  40.2  per 
cent  (51)  were  statistically  significant.  This  general  trend  supports 
arguments  favoring  the  fidelity  of  the  exercises  for  measuring  the 
dimensions  involved.  Each  dimension  is  treated  in  more  detail  be- 
low. 

The  Pearson  correlation  coefficients  among  the  scores  de- 
signed to  reflect  specifically  the  learning  dimension  are  given  in 
Table  11.  All  but  7 of  the  28  correlation  coefficients  of  Table  11 
are  statistically  significant.  These  statistically  significant  corre- 
lation coefficients  range  from  . 17  to  . 80,  with  an  interquartile 
range  from  . 22  to  . 43.  These  correlation  coefficients  indicate 
that  the  learning  oriented  tests  are,  by  and  large,  moderately  re- 
lated. This  finding  was  expected  because  the  exercises  range  in 
form  from  a conceptual  troubleshooting  task  to  a timed  inspection 
test,  to  a timesharing  task  including  a pipe  assembly  and  a control 
watch.  The  high  percentage  of  statistically  significant  intercorre- 
lations can  be  taken  as  an  indication  or  confirmation  of  the  ability 
of  these  exercises  to  measure  various  aspects  of  a common,  al- 
beit complex,  dimension. 
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Table  1 1 


The  Pearson  product  moment  correlations  among  the  scores 


designed  to  measure  the  psychophysical  and  motor  dimension  are 
given  in  Table  12.  All  but  one  of  these  scores  was  moderately  re- 
lated with  the  other  scores.  Some  accuracy  scores  on  the  Inspec- 
tion/Sort exercise  were  negatively  correlated  with  scores  that  re- 
flect speed  in  the  Inspection/ Sort  exercise.  This  tradeoff  between 
speed  and  accuracy  is  not  unusual. 

Of  the  21  correlation  coefficients  between  these  scores,  14 
(66.  7 per  cent)  are  significant.  This  seems  considerably  higher 
than  the  20.  3 per  cent  significant  correlation  coefficients  in  inter- 
dimensional  correlations,  as  described  in  a previous  section.  The 
range  of  statistically  significant  correlation  coefficients  is  from  .20 
to  . 98.  The  interquartile  range  is  .23  to  . 57. 

These  moderate,  positive  correlations  can  be  construed  as 
support  for  the  conjecture  that  these  exercise  scores  reflect  as- 
pects of  one  underlying  dimension. 

Social/ Motivational  Dimension 

The  intercorrelations  among  the  scores  designed  to  consider 
the  Social/ Motivational  dimension  are  given  in  Table  13.  For  the  16 
correlation  coefficients  that  differed  significantly  from  zero,  11  are 
positive  and  6 are  negative.  The  negative  correlations  all  reflect 
differences  in  scores  on  the  Level  of  Aspiration  exercise.  The  pes- 
simism scores  correlated  negatively  with  some  of  the  other  scores 
(the  first  goal  attainment  level  estimate  and  the  optimism  rating). 
There  was  also  a negative  correlation  between  the  realism  score 
and  the  rating  of  optimism.  Perhaps  these  scores  cannot  vary  in- 
dependently in  a task  such  as  this  one;  that  is,  a highly  optimistic 
estimate  cannot  be  realistic  or  an  individual  cannot  be  optimistic 
and  pessimistic  at  the  same  time. 

The  overall  low  level  of  interrelationship  between  some  of 
the  scores  on  the  Level  of  Aspiration  exercise  and  the  other  meas- 
ures of  this  dimension  reflects  the  uniqueness  for  these  measure- 
ments. Of  the  78  correlations  within  the  Social/ Motivational  di- 
mension, 16  (20.  5 per  cent)  are  statistically  significant.  This  val- 
ue may  be  compared  with  20.  3 per  cent  significant  correlation  co- 
efficients among  all  the  interdimensional  correlations.  This  mod- 
erate level  of  agreement  within  these  scores  may  be  taken  as  sup- 
port for  the  conjecture  that  these  measures  involve  a common  con- 
struct. 
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Intercorrelations  Among  Scores  Relevant  to  Social/ Motivational  Dimension 


Signs  of  some  correlation  coefficients  changed  so  that  positive  relationships  indicate  increasing  magnitude  on 


Policy  Capturing  of  Dimensional  Ratings 

One  of  the  intermediate  steps  in  the  classification  decision 
making  process  was  rating  each  man  relative  to  each  of  the  three 
dimensions  considered  by  the  job  analysis.  One  administrator  rat- 
ed each  man.  The  policy  of  the  individual  Center  administrators 
and  the  overall  policy  of  the  administrators  were  investigated.  The 
goal  of  these  policy  capturing  analyses  was  to  determine  the  extent 
to  which  the  ratings  of  the  Center  can  be  seen  to  be  themselves  pre- 
dicted by  a weighted  sum  of  individual  test  scores.  Stated  alterna- 
tively, the  policy  capturing  approach  seeks  to  establish  the  extent 
to  which  the  tests  captured  the  decision  making  policy  of  the  admin- 
istrators. To  this  end,  a stepwise  multiple  linear  regression  anal- 
ysis procedure  was  used.  This  procedure  allows  the  evaluation  of 
each  score  relative  to  the  decisioti  of  the  administrator(s). 

The  policy  capturing  approach  to  decision  analysis  possesses 
a fairly  recent  history  of  application.  Madden  (1964),  Stephenson  and 
Ward  (1971).  Bottenberg  and  Christal  (1 968 ),  and  Christal  (1968a,  1968b, 
1963)  described  its  employment  for  Air  Force  officer  advancement 
evaluative  purposes,  and  Siegel  and  Federman  (in  press)  employed  the 
approach  for  deriving  emphasis  areas  in  Air  Force  Technical  Train- 
ing. The  approach  has  also  men  applied  inawide  variety  of  other 
areas  including,  but  not  limited  to,  judgments  of  personality  charac- 
teristics (Hammond,  Hursch,  & Todd,  1964),  attraction  of  common 
stocks  (Slovic,  1969),  mental  illness  diagnosis  (Goldberg,  1970), 
and  judgments  of  admissibility  to  graduate  school  (Dawes,  1970, 

1971).  While  there  is  some  theoretic  controversy  relative  to  the 
use  of  the  additive  model  for  such  work,  Slovic  and  Lichtenstein 
(1970)  after  a comprehensive  review  of  studies  employing  the  linear 
approach,  concluded: 

In  all  of  these  situations  the  linear  model  has  done 
a fairly  good  job  of  predicting  the  judgments,  as  indi- 
cated by  rg  values  in  the  .80s  and  .90s  for  the  artifi- 
cial tasks  and  the  .70s  for  the  more  complex  real-world 
situations  (p.36). 

If  acceptable  multiple  correlation  coefficients  are  evidenced, 
then  it  can  be  said  that  the  various  scores  are  actually  reflective  of 
the  policy  employed  by  the  administrator(s)  in  making  a rating. 


J 
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The  regression  equation  gives  the  weights  seemingly  used 
by  the  rater.  The  square  of  the  multiple  correlation  coefficient 
gives  the  proportion  of  variance  the  policy  accounts  for.  In  the 
analyses  here  completed,  only  the  original  scores  were  used.  No 
attempt  was  made  to  introduce  nonlinear  terms  into  the  regression 
equation  by  generating  additional  scores  from  the  raw  scores.  Of 
course,  from  a theoretical  point  of  view,  the  ability  to  predict  a 
decision  using  a machematical  formula  does  not  mean  that  the  actu- 
al decision  making  process  has  been  identified.  However,  from  a 
pragmatic  point  of  view,  the  decision  can  be  said  to  be  understood 
and  captured  by  such  a formula. 

Policy  in  Rating  Men  on  Learning  Dimension 

Following  Cristal  (1963),  the  individual  policies  of  the  five 
administrators  should  all  be  more  fully  captured  by  a multiple  re- 
gression approach  than  the  one  overall  policy  of  the  five  adminis- 
trators considered  together.  The  policies  of  the  individual  admin- 
istrators were  indeed  captured  to  a greater  extent  for  the  learning 
dimension  than  the  group  policy.  However,  for  three  of  the  admin- 
istrators, the  N was  small  relative  to  the  number  of  tests.  The 
multiple  R for  these  administrators  must  be  interpreted  according- 
ly. Stepwise  multiple  regression  analyses  reflecting  the  policies 
of  the  five  individual  administrators  were  conducted  to  reveal  the 
tests  most  emphasized  by  each  administrator,  relative  to  the  learn- 
ing dimension. 

The  multiple  correlation  coefficients  for  the  five  adminis- 
trators were:  1.  00,  1.  00,  . 95,  . 91,  and  . 85.  These  indicate  a 
very  considerable  predictiveness  of  the  rating  by  the  exercise 
scores.  Additionally,  if  the  multiple  correlation  can  be  consider- 
ed to  represent  valid  covariance,  then  these  values  provide  some 
measure  of  intrarater  (rate-rerate)  reliability.  Moreover,  this 
rate-rerate  reliability  measure  is  derived  without  the  negative  ef- 
fects of  an  administrator  rating  the  same  person  twice. 

The  extent  to  which  there  is  agreement  in  policy  relative 
to  judgments  on  the  learning  dimension  among  the  administrators 
is  indicated  by  the  extent  to  which  the  grouped  data  of  these  five 
administrators  can  be  described  by  one  policy. 
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The  overall  group  policy  concerning  the  learning  dimension 
was  evaluated  through  a stepwise  multiple  linear  regression  analy- 
sis. The  group  policy  emphasized  the  Conceptual  Integration/ Ap- 
plication; Tool  and  Object  Nomenclature,  Use,  and  Recognition; 
and  the  Dual  Task  construction  scores.  The  other  scores  did  not 
contribute  substantially  to  the  explanation  of  the  group  rating  pol- 
icy on  this  dimension.  The  analysis  is  summarized  in  Table  14. 
The  very  high  multiple  correlation  obtained  for  this  dimension  (R  = 

. 114)  may  be  interpreted,  according  to  the  literature,  as  indicating 
high  agreement  among  the  policies  of  the  individual  members  of  the 
TCAC  staff  (Christal,  1963). 


Table  14 


Stepwise  Zero 

Multiple  Increase  Order 


Step 

Score 

R 

R 2 

in  R‘ 

r 

1 

Conceptual  Integration /Application 

.71 

. 51 

vA 

LO 

1 

.71 

2 

Tool-Object  Nomenclature,  Use, 
and  Recognition 

.78 

. 61 

.10 

. 60 

3 

Dual  Task:  construction  score 

.82 

. 67 

.05 

.41 

4 

Interview:  score  2 

CO 

CO 

00 

.01 

-.25 

5 

Inspection/Sort:  speed  and 
accuracy  score 

CO 

CO 

. 69 

.01 

.12 

6 

Inspection/Sort:  accuracy  score 

.83 

. 69 

0* 

.15 

Multiple  R based  on  Center  test  saoves=  .84 
*t38s  than  .005 


The  group  policy  is  also  summarized  in  Table  15  for  ease 
of  comparison  with  the  policies  of  the  individual  administrators. 
Table  15  indicates  that  two  of  the  three  tests  most  emphasized  by 
each  individual  policy  coincide  with  two  of  the  three  tests  most  em- 
phasized by  the  group  policy. 
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Relative  (Order  of  Regression  Entry)  Importance  of  Center  Scores  Most  Weighted 
in  the  Policies  of  Each  Administrator  and  of  the  Administrators  as  a Group 


The  exercise  most  emphasized  by  the  policy  of  each  admin- 
istrator and  the  three  exercises  most  emphasized  by  the  group  pol- 
icy were  all  designed  to  assess  the  learning  dimension.  However, 
some  exercises  designed  to  tap  other  dimensions  were  used  in  the 
learning  dimension  policies  of  both  the  individuals  and  the  group. 

For  example,  for  administrator  number  1,  the  score  entered  third 
into  the  stepwise  multiple  regression  analysis  (and,  accordingly, 
emphasized  more  than  all  the  othe  ' scores  but  two)  was  a Level  of 
Aspiration  score.  The  Level  of  Aspiration  score  was  designed  to 
assess  the  Social/ Motivational  dimension.  Other  social/motiva- 
tional scores  were  included  by  other  administrators.  The  inclu- 
sion of  social  oriented  test  scores  in  the  policy  for  the  learning 
dimension  might  be  an  indication  of  a "halo  effect.  " The  halo  ef- 
fect in  this  case  would  be  an  unwarranted  carryover  of  a high  rat- 
ing in  one  dimension  to  the  rating  of  another  dimension.  However, 
analysis  of  the  social  related  test  scores  included  in  the  policy  rel- 
ative to  the  learning  dimension  indicated  that  the  weighting  of  these 
scores  w s small  and  sometimes  negative.  Further,  the  overall 
correlation  of  the  ratings  on  the  learning  and  social  dimensions 
was  . 27,  which  indicates  only  a modest  relationship  between  the 
two  ratings. 

Policy  in  Rating  Men  on  Psychophysical-Motor  Dimension 

The  individual  multiple  R values  for  administrator  policy 
capturing  regression  analyses  relative  to  the  psychophysical-motor 
dimension  were;  1.  00,  1.  00,  . 92,  . 91  and  . 82.  Following  the  log- 
ic presented  earlier,  this  suggests  acceptable  intrarater  agreement. 

The  combined  policy  of  all  administrators  in  the  psychophy- 
sical-motor dimension  was  investigated  in  the  same  manner  as  for 
the  learning  dimension.  The  results  of  this  analysis  indicated  that 
the  group  policy  emphasized  the  Dual  Task  construction  score  and 
a Coordinative  Speed  and  Accuracy  score  (Table  16).  All  of  the 
four  most  heavily  weighted  scores  were  designed  to  evaluate  this 
dimension.  The  obtained  multiple  R was  . 77. 

The  policies  for  the  individual  administrators  were,  as  ex- 
pected, more  fully  captured  than  the  group  policy.  The  individual 
policies  are  summarized  in  Table  15.  The  score  most  emphasized 
by  each  administrator  was  one  of  the  two  most  emphasized  by  the 
group  policy  (described  above)  and  was  a test  designed  to  assess 
the  psychophysical  and  motor  dimension. 
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Table  16 


Group  Policy  for  Hating  Men  on  the  Psychophysical-Motor  Dimension 


Step 

Score 

Stepwise 
Mul tiple 
R 

R2 

Increase 
in  R‘ 

Zero 

Order 

r 

1 

Dual  Task:  construction  score 

.61 

.38 

.38 

. 61 

o 

Coordinative  Speed  and  Accuracy: 
speed  and  accuracy  score 

.69 

.48 

.10 

. 56 

3 

Inspection-Sort:  speed  and 
accuracy  score 

.73 

. 53 

.05 

.40 

4 

Coordinative  Speed  and  Accuracy: 
accuracy  score 

.74 

.55 

.02 

.39 

5 

Reliability 

.75 

. 56 

. 01 

-.20 

6 

Interview:  mean  rating 

.75 

. 56 

.01 

. 03 

Multiple  R based  on  optimum  combination  of  scores = .77 


However,  the  policies  of  both  the  group  and  the  individuals 
included  some  emphasis  on  scores  intended  to  assess  other  dimen- 
sions. The  extent  of  this  cross-dimensional  influence  can  be  esti- 
mated by  the  correlation  between  the  psychophysical  and  motor  di- 
mension and  the  other  dimensions.  These  correlations,  as  stated 
above,  are  .42  and  .25  for  the  learning  and  the  social/ motivational 
dimensions,  respectively.  These  are  modest  levels  and  indicate  a 
relatively  large  independence  of  these  dimensions. 

Policy  in  Hating  Men  on  Social/ Motivational  Dimension 

The  combined  policy  of  the  TCAC  administrators  on  the  soci- 
al/motivational dimension  was  less  uniform  than  the  combined  pol- 
icies on  the  other  two  dimensions.  There  was  moderately  high  con- 
sistency in  the  policy  of  each  administrator  on  the  social/  motivation- 
al dimension.  The  multiple  Hs  indexing  this  are:  1.  00,  1.  00,  . 85, 

. 77.  and  .77.  The  analysis  >s  summarized  in  Table  15. 

The  two  scores  most  emphasized  by  the  policies  of  the  indi- 
vidual administrators  were  not  designed  to  assess  the  social/moti- 
vational  dimension.  This  indicates  that  the  administrators  had  some 
difficulty  in  making  these  ratings  using  the  exercise  scores.  How- 
ever, the  overlap  of  the  ratings  of  this  dimension  with  those  of  the 
other  twfo  dimensions  was  low,  as  indicated  by  the  correlation  co- 
efficients of  . 27  and  . 25  with  the  learning  and  the  psychophysical 
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and  motor  dimensions  respectively.  The  overall  agreement  of 
the  individual  policies  is  reflected  in  the  multiple  R between  the 
test  scores  and  the  ratings  of  all  administrators.  This  multiple 
R was  . 66.  The  three  tests  most  emphasized  by  the  group  policy 
were  all  social/ motivational  related  tests:  the  Group  task,  the  In- 
terview mean  score,  and  the  Reliability  exercise.  This  analysis 
is  summarized  in  Table  17. 


Table  17 

Group  Policy  for  Rating  Men  on  the  Social/ Motivational  Dimension 


Stepwi se  Zero 


Step 

Score 

Multipl 

R 

R2 

Increase 
in  R- 

Order 

r 

1 

droup  Task 

. 51 

.26 

.26 

. 51 

o 

Interview:  mean  score 

. 56 

.31 

.05 

.28 

3 

Reliability 

.60 

.36 

.05 

-.26 

4 

Coordinative  Speec,  and  Accuracy: 
accuracy  score 

.62 

.38 

.02 

.27 

5 

Inspection/Sor  t : accuracy  score 

.63 

. 39 

.01 

-.12 

6 

Concer tual  Integration/Application 

.63 

.40 

.01 

.20 

Multiple  R based  on  optimum  combination 

of  Center 

test  score s=  .66 

TCAC  Career  Field  Recommendations 

The  purpose  of  the  TCAC  procedures  was  to  reclassify  low 
aptitude  enlistees  and,  where  possible,  to  identify  career  fields  in 
which  they  might  do  well.  The  resulting  classifications  were  sum- 
marized earlier  (Table  6). 

The  career  field  recommendations  of  the  Center  were  based 
on  a vote  which  followed  a group  discussion  in  which  all  data  for  an 
individual  were  evaluated.  The  extent  to  which  the  career  field 
recommendations  of  the  TCAC  can  be  predicted  directly  from  the 
various  scores  is  of  interest.  This  form  of  policy  capturing  has 


been  called  "paramorphic  representation"  of  the  judgmental  proc- 
ess by  a mathematical  model  (Dawes,  1971).  This  approach  pos- 
sesses important  implications  for  large  scale  application  of  the 
TCAC  method  because  it  investigates  the  potential  of  replacing  the 
group  decision  making  process  by  its  mathematical  representation. 
This  potential  approach  has  also  been  termed  "bootstrapping.  " Quite 
suprisingly,  when  criterion  data  are  available,  this  technique  has,  in 
a variety  of  situations,  been  shown  to  be  more  effective  (valid)  than 
the  performance  of  the  judges  on  which  the  bootstrapping  model  is 
based.  Alternative  explanations  for  this  emphasize  either:  (a)  the 
ability  of  the  model  to  detect  the  true  policy  and  reject  unwanted 
noise,  and/or  (b)  the  relative  robustness  of  linear  models  (Dawes 
& Corrigan,  1974). 

The  extent  to  which  the  career  field  recommendations  can 
be  predicted  by  the  various  scores  was  investigated  using  a dis- 
criminant analysis  technique-.  This  technique  can  be  used  to  clas- 
sify individuals  based  on  their  relative  similarity  to  each  of  two 
or  more  subgroups.-  in  the  current  case,  the  actual  classification 
recommendation  for  general  rates  or  for  a specific  rating.  The 
analysis  indicated  that  the  means  of  the  various  test  scores  of 
individuals  receiving  the  two  types  of  recommendations  were  sig- 
nificantly different  (generalized  Mahalanobis'  D^  = 71.  3,  which  is 
distributed  as  chi  square  with  16  d.  f. , p < . 05).  The  results  of  the 
classification  analysis  are  summarized  in  Table  18.  The  classifi- 
cations based  on  this  analysis  are  78.  6 percent  accurate.  A linear 
function  of  the  test  scores  thus  captures  well  the  policy  used  by  the 
TCAC  administrators  in  making  career  field  recommendations. 

Based  on  the  discriminant  analysis,  it  appears  that  the  bootstrap- 
ping approach  holds  promise  for  considerably  streamlining  the 
TCAC  procedures. 
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Table  lb 


Comparison  of  Actual  and  Statistical  Classifications 

of  Men 

Actual  Hecommendation 

Total 

Discriminant 

General 

Specific 

No.  of 

Analysis 

Hates 

Rating 

Men 

General 

Hates 

47 

13 

64 

Specific 

Hating 

17 

63 

76 

140 

Generalized  M 
o ith  16  1.  f. 

ahalanobis'  D^=  71.3,  } 

< .05  (distributed  a 

s ihi  . • ■ - 

ra  nt  igt  correct  classifications=  78.6. 
Prediction  of  On-the-Job  Success 


The  decisions  of  the  TCAC  concerning  the  level  of  success 
expected  in  the  Fleet  were  clinical  (subjective)  in  nature.  These 
decisions  were,  however,  based  on  and  cast  against  the  backdrop 
of  the  data  which  evolved  from  the  various  procedures.  The  ap- 
proach applied  in  this  analysis  attempts  to  make  explicit,  through 
multiple  regression  analysis,  those  tests  emphasized  by  the  TCAC 
administrators  in  reaching  decisions  concerning  predicted  level  of 
success  in  the  field.  Additionally,  the  utility  of  the  Navy  test 
scores  in  explaining  (i.  e.  , capturing  the  policy  of)  the  TCAC  pre- 
dictions of  job  success  was  investigated. 

Initially,  predictions  of  job  success  were  thought  to  be  de- 
pendent on  the  preceding  career  field  recommendation  made  by  the 
Center.  As  described  above,  the  discriminant  analysis  indicated 
that  the  two  subgroups  differed  significantly  with  respect  to  the 
mean  scores  on  the  TCAC  exercises.  Therefore,  analyses  were 
completed  separately  for  men  recommended  for  general  rates  and 
men  recommended  for  machinist's  mate.  Only  these  two  subsets 
had  a sufficiently  large  number  to  justify  analysis.  Additionally, 
an  analysis  of  the  total  sample  was  completed. 
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Group  Policy  for  Machinist's  Mate  and  General  Rates  Subsamples 

Two  multiple  linear  regression  analyses  identified  the  linear 
combination  of  TCAC  test  scores  that  best  predicted  the  Center  pre- 
dictions of  on-the-job  performance  for  two  subsamples  of  men;  those 
recommended  for  machinist's  mate  (N  = 38),  and  those  recommended 
for  general  rates  (N  = 64).  The  six  scores  that  together  accounted 
for  the  greatest  amount  of  variance  in  the  predicted  level  of  success 
for  each  of  these  two  subgroups  are  given  in  Table  19  for  the  ma- 
chinist's mate  subsample  and  in  Table  20  for  the  general  rates  sub- 
sample. For  the  machinist's  mate  subsample,  the  multiple  corre- 
lation, based  on  16  center  measures,  was  . 90.  For  the  general 
rates  subsample,  the  multiple  correlation,  based  on  16  center 
measures,  was  . 70.  The  basic  interpretation  of  these  two  analyses 
is  that  the  policy  of  the  Center  in  predicting  Fleet  job  success  was 
substantially  "captured"  for  both  subsamples. 
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Table  19 

Policy  Capturing  of  Level  of  Job  Success  in  the  Fleet  by  Score-- 


Machinist's  Mate  Subsample  (N 

= 38) 

Step 

Score 

Stepwise 

Multiple 

R 

R2 

Increase 
in  R2 

Zero 

Order 

r 

1 

Reliability 

. 53 

. 28 

.28 

-.53 

2 

Conceptual  Integration/ Application 

.65 

.42 

.14 

. 50 

3 

Coordinative  Speed  and  Accuracy: 
speed  score 

.72 

.52 

. 10 

.29 

4 

Group  Task 

.77 

. 59 

.07 

.36 

5 

Tool  and  Object  Nomenclature, 
Use,  and  Recognition 

.79 

.63 

. 04 

.09 

6 

Coordinative  Speed  and  Accuracy: 
accuracy  score 

.81 

.65 

.02 

.37 

Multiple  R based  on  optimum  combination  of  saores=  .90 
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Table  20 


Policy  Capturing  of  Level  of  Job  Success  in  the  Fleet  by  Score-- 
General  Hates  Subsample  (N  = 64) 


Step 

Score 

Stepwise 

Multiple 

R 

R2 

Increase 
in  R2 

Zero 

Order 

r 

Conceptual  Integ 

rat ion/Applicat ion 

.47 

.22 

.22 

. 47 

2 

Reliability 

. 57 

. 32 

.10 

-.25 

3 

Interview:  mean 

score 

. 64 

.40 

.08 

.31 

4 

Dual  Task:  construction  score 

.66 

.43 

.03 

-.10 

5 

Inspection/Sort : 

accuracy  - 

rood  items 

. 67 

.45 

. 01 

-.22 

6 

I ns sect ion /Sort : 

speed  and 

accuracy 

.68 

.46 

.01 

.05 

• . . • 

t ie  P.  bised  on  op 

t imum  combi na tion 

of  scores= 

.70 

The  Reliability  and  the  Conceptual  Integration/Application 
scores  were  identified  by  these  analyses  as  the  most  important 
factors  in  decisions  concerning  both  subsamples.  The  emphasis 
of  the  administrators  on  the  Reliability  test  can  be  further  explain- 
ed on  the  basis  of  observations  made  at  meetings  at  which  these 
predictions  and  the  recommendation  of  ratings  were  made.  It  ap- 
pears that  the  Reliability  exercise  score  was  used  as  a "killer" 
item.  The  staff  generally  thought  poorly  of  those  men  who  did 
poorly  on  the  Reliability  exercise.  These  men  were,  generally, 
not  recommended  to  specific  ratings.  However,  two  men  recom- 
mended to  be  machinist's  mates  failed  this  test.  They  possessed, 
presumably,  other  positive  attributes  that  caused  the  staff  to  rec- 
ommend them.  These  men  were  predicted  to  have  only  minimally 
acceptable  level  of  on-the-job  success. 
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The  second  exercise  identified  as  a primary  constituent 
of  the  prediction  policy,  the  Conceptual  Integration/ Application, 
involves  a complex  troubleshooting  task.  This  exercise  was  de- 
signed to  assess  the  learning  dimension.  This  score  was  most 
highly  correlated  with  level  of  success  predictions  for  the  gen- 
eral rates  subsample  (zero  order  correlation  » . 47).  It  was  the 
second  most  highly  correlated  score  with  the  level  of  success  pre- 
diction for  the  machinist's  mate  subsample  (zero  order  correla- 
tion = . 50).  We  note  that  the  two  subsamples  differed  significant- 
ly in  mean  level  of  performance  on  this  test  (t  = 3.  0,  p < . 05). 

This  exercise  score,  then,  was  an  important  part  of  the  policy  of 
the  TCAC  staff's  level  of  on-the-job  success  predictions  within 
both  of  these  subgroups. 

For  both  subgroups,  these  two  exercises  (Reliability  and 
Conceptual  Integration/ Application)  largely  describe  the  policy  of 
the  administrators  with  respect  to  the  predicted  level  of  job  suc- 
cess within  the  two  subgroups  involved.  The  other  scores  added 
incrementally  and  significantly.  However,  no  single  test  alone 
stands  out  as  particularly  or  predominantly  important  in  the  for- 
mation of  the  rating  policy.  There  was  no  overlap  between  the 
subgroups  in  the  next  four  tests  identified  by  the  multiple  regres- 
sion analysis  as  adding  most  to  the  predictable  variance  of  level 
of  success.  For  each  subgroup,  the  first  six  tests  that  are  iden- 
tified by  the  multiple  regression  analysis  include  tests  designed 
to  assess  each  of  the  three  dimensions  included. 

Group  Policy  With  Respect  to  the  Total  Sample 

The  predictions  of  level  of  job  success  may  be  expected 
to  be  highly  dependent  on  the  prior  Center  recommendation  of  a 
specific  career  field.  However,  it  is  heuristic  to  assess  the  Cen- 
ter's policy  in  making  these  predictions  for  the  total  sample,  dis- 
regarding temporarily  the  previously  mentioned  problems  intro- 
duced by  career  field  considerations. 

A stepwise  multiple  linear  regression  analysis  identified 
the  linear  combination  of  scores  that  best  captured  the  staff's 
policy  for  the  total  group.  The  six  scores  that  together  accounted 
for  the  highest  amount  of  variance  in  predicted  scores  are  given 
in  Table  21.  Tests  designed  to  assess  all  three  dimensions  are 
represented  in  these  six  tests.  The  complex  Conceptual  Integra- 
tion/Application score  was  most  heavily  weighted.  This  emphasis 
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reflects  the  staff's  emphasis  on  the  learning  dimension.  The  Re- 
liability score  was  next  most  highly  influential  in  predicting  the 
on-the-job  success.  This  exercise  reflects  the  social/ motivation- 
al dimension. 


Table  21 

Policy  Capturing  of  Level  of  Job  Success  in  the  Fleet  byScore-- 


Total  Sample  (N  - 140) 


Stepwise 

Multiple 

Increase 

Zero 

Step 

Score 

R 

R2 

in  R2 

Order  r 

l 

Conceptual  Integration/ 

Application 

.47 

.22 

.22 

. 47 

2 

Reliability 

. 58 

.23 

. 11 

-.33 

1 3 

Interview:  mean  rating 

.61 

.37 

. 04 

.27 

4 

Croup  Task:  total  score 

.63 

.40 

.02 

.24 

5 

Coordinative  Speed  and 

Accuracy:  speed  score 

.64 

.41 

.01 

.27 

6 

Inspection/Sort:  accuracy 

score 

.65 

.42 

.01 

-.05 

Mu.  t'll 

Ze  R based  on  optimum  oombina 

t ion  of  Si 

?ores= 

.68 

Overall,  the  six  most  important  test  scores  identified  by 
the  multiple  regression  analysis  accounted  for  42  per  cent  of  the 
variance  of  the  administrator's  predictions  of  fleet  job  success 
(multiple  R - . 68).  If  one  interprets  the  group  decision  as  a cri- 
terion, this  level  of  prediction  is  quite  impressive  compared  to 
other  predictive  validity  studies. 
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The  policy  concerning  the  total  sample  can  be  compared  to 
the  policies  for  each  of  the  two  subgroups  previously  discussed, 
men  recommended  for  machinist's  mate  and  men  recommended 
for  general  rates.  This  comparison  can  be  made  first  concern- 
ing the  extent  of  policy  capturing,  and  second  concerning  the  tests 
identified  by  the  multiple  regression  analysis  as  important.  As 
would  be  anticipated,  the  policy  for  the  total  group  overlaps  con- 
siderably with  the  policies  of  the  two  subgroup  constituent  parts 
of  the  total  group.  The  two  exercises  most  emphasized  in  the 
three  policies  were  identical;  the  Conceptual  Integration/ Appli- 
cation and  the  Reliability  tests.  The  remaining  four  of  the  six 
most  influential  scores  in  the  whole  group  policy  include  two  tests 
emphasized  by  each  of  the  two  subgroup  policies.  In  all,  the  six 
tests  most  emphasized  by  the  total  group  policy  included  tests 
from  each  of  the  three  dimensions  identified  by  the  job  analysis. 

The  extent  to  which  the  variance  of  the  policy  was  cap- 
tured, as  indexed  by  the  multiple  R based  on  all  16  Center  meas- 
ures, is  .68  for  the  whole  group,  . 90  for  the  machinist's  mate 
subgroup,  and  . 70  for  the  general  rates  subgroup.  The  shrinkage 
in  the  multiple  R from  the  machinist's  mate  subgroup  to  the  whole 
sample  is  understandable  in  terms  of;  (a)  the  expected  more  strin- 
gent requirements  set  by  the  Center  for  a given  level  of  success 
prediction  for  men  recommended  for  more  demanding  ratings,  and 
(b)  possible  variations  in  policy  for  the  eight  types  of  career  field 
recommendation  subgroups  depending  upon  the  dimensional  require- 
ment of  the  specific  ratings. 

Comparison  of  Navy  and  TCAC  Scores 

Analyses  were  conducted  to  determine  the  extent  to  which 
the  Navy  test  scores  (GCT,  ARI,  MECH,  CLER,  ETST,  and  SHOP  PR) 
could  be  used  to  explain  (capture  the  policy  of)  the  level  of  the  job 
success  predictions.  These  scores  were  subjected  to  the  stepwise 
multiple  regression  analyses  as  reported  above  for  the  Center  ex- 
ercise scores.  The  multiple  correlation  coefficients  obtained  were 
. 12,  .41,  and  .23  for  the  general  rates  subgroup,  machinist's  mate 
subgroup,  and  the  whole  sample  respectively.  The  corresponding 
multiple  Rs  reported  above  based  on  six  Center  test  scores  are,  re- 
spectively, . 68,  . 81,  and  . 65.  Accordingly,  the  Navy  tests  can  be 
seen  to  have  relatively  little  utility  in  capturing  the  policy  of  the 
TCAC  relative  to  level  of  success  predictions.  Further,  their  util- 
ity varies  relatively  greatly  from  subgroup  to  subgroup,  and  from 
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subgroups  to  the  whole  sample.  This  is  quite  understandable  as 
the  TCAC  exercises  were  designed  to  be  the  basis  for  the  staff's 
decisions.  These  comparisons  further  indicate  that  the  decisions 
of  the  TCAC  administrators  were  based  on  the  Center  exercise 
scores  and  that  these  scores  are  not  redundant  with  the  available 
Navy  test  scores.  These  analyses  are  summarized  in  Tables  22, 
2 3,  and  24. 


Table  22 

Policy  Capturing  of  Level  of  Job  Success  in  the  Fleet  by  Navy  Test  Score- 

General  Rates  Subsample  (N  - 64) 


Step 


Navy  Test 


SHOP 

MECH 

ETST 


Stepwise 
Multiple  R 


.09 

.11 

.12 


.01 
.01 
. 01 


Increase 
in  R2 


.01 

0* 

0* 


Zero 
Order  r 


.09 

.03 

.02 


Multiple  R based  on  optimal  combination  of  Navy  test  scores-  .12 
*less  than  .005 


Table  24 

Policy  Capturing  of  Level  of  Job  Success  in  the  Fleet  by  Navy  Test  Scores- - 
Total  Sample  (N  = 140) 


Step 

Navy  Test 

Stepwise 
Multiple  R 

R2 

Increase 
in  R 2 

Zero 

Order 

1 

SHOP  PR 

.16 

.03 

. 03 

.16 

2 

ETST 

.19 

.03 

.01 

-.05 

3 

MECH 

.21 

.04 

.01 

.14 

4 

CLER 

.22 

.05 

0* 

.04 

5 

ARI 

.22 

.05 

0* 

.02 

6 

GCT 

.23 

.05 

0* 

.02 

Multiple  R based  on  optimal  combination  of  Navy  test  scores ~ .23 
*less  than  .005 


Views  of  Individuals  Assessed 

The  opinions  of  the  individuals  assessed  in  the  TCAC  rela- 
tive to  the  various  exercises  were  systematically  elicited  through 
a questionnaire  which  contained  both  multiple  choice  and  open-end- 
ed questions.  The  questionnaire  was  administered  at  the  conclu- 
sion of  the  assessment  day. 

The  multiple  choice  data  concerning  the  perceived  fairness 
indicated  unanimous  agreement  that  the  TCAC  exercises  were  fair 
in  an  absolute  sense.  In  a choice  between  "yes"  and  "no,  " indicat- 
ing agreement  or  disagreement  wilh  a statement  that  the  TCAC 
tests  seem  fair,  100  per  cent  of  the  sample  responded  "yes.  " This 
topic  was  further  probed  by  asking  the  individuals  who  participated 
in  the  assessment  process  to  compare  the  TCAC  exercises  with 
other  tests  taken  in  the  past.  Based  on  a five  category  Likert  type 
scale,  31  per  cent  of  the  enlistees  considered  the  TCAC  exercise 
tests  "very  much  more  fair.  " 42  per  cent  found  the  TCAC  exercises 
"more  fair,  " and  26  per  cent  said  the  TCAC  exercises  were  "equally 
fair.  " Only  one  individual  (less  than  1 per  cent  of  the  sample) chose 
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the  "less  fair"  category.  No  one  chose  the  "very  much  less  fair"  cate- 
gory. There  was  also  unanimous  agreement  that  the  TCAC  exercises 
were  enjoyable.  On  a four  category  scale,  76  per  cent  of  the  individu- 
als assessed  chose  the  superlative  "enjoyed  very  much"  response.  No 
one  chose  the  nonenjoyable  category. 

Reasons  for  Fairness  Evaluation 

The  reasons  for  perceiving  the  exercises  to  be  fair  were  in- 
quired into  through  a completion  type  question.  Typical  replies  to 
this  question  are  listed  below.  Reasons  frequently  given  for  thinking 
the  TCAC  exercises  were  fair  fell  in  two  major  areas:  (a)  the  training 
completeness  and  hands  on  aspects  of  the  learning  phase  of  the  exer- 
cises, and  (b)  the  exercises  emphasized  performance,  not  reading  and 
w riting. 

"Gave  me  a chance  to  prove  that  I could  do  some  things  with  my 
hands,  not  just  my  head." 

"Very  fair.  It  was  explained  very  well  every  time,  to  make  sure 
I understood  what  was  going  on,  and  I did.  " 

"Yes.  Because  these  weren't  tricky  like  others.  " 

"it  gives  a person  an  opportunity  to  use  his  hands  in  a certain  situ- 
ation, instead  of  learning  it  from  a book.  Like  some  tests  you  have 
to  read  questions  and  a certain  amount  of  time  is  alloted  (sic)  for  the 
test.  Some  people  can't  read  fast  enough  and  that  isn't  fair.  But 
here  you  can  use  your  hands.  " 

"Because  it  was  a lot  less  writing  and  more  doing.  " 
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Opinions  Concerning  Specific  Exercises 


The  questionnaire  asked  the  individuals  who  were  assess- 
ed to  indicate  (using  a checkoff  list)  the  two  exercises  they  "liked 
most"  and  the  two  they  "liked  least.  " A number  of  times  the  per- 
sons processed  volunteered  that  there  were  no  tests  that  they  "dis- 
liked" but  that  they  chose  the  two  tests  that  they  liked  least  to  re- 
spond to  the  question.  The  three  exercises  liked  best  were  the: 
Dual  Task,  Conceptual  Integration/ Application,  and  the  Coordina- 
tive  Speed  and  Accuracy.  These  were  all  job  learning  perform- 
ance exercises.  This  agrees  with  the  stated  opinions  that  the  ex- 
ercises were  appreciated  which  taught  requisite  skills  and  allowed 
performance  of  actual  tasks.  Additionally,  there  was  some  em- 
phasis in  response  to  this  question  relative  to  the  "challenge"  of- 
fered by  these  exercises.  The  number  of  individuals  choosing  a 
test  as  "liked  best"  correlated  negatively  (rho  = . 40)  with  the  num- 
ber of  enlistees  who  chose  that  test  as  "liked  least.  " The  three 
exercises  liked  least  were:  Reliability,  Level  of  Aspiration,  and 
Group  Task.  There  was  no  one  unifying  reason  for  these  being 
"liked  least.  " However,  relative  to  the  Group  Task,  several  per- 
sons mentioned  that  they  did  not  like  performing  in  a group  before 
they  knew  the  other  group  members. 


DISCUSSION  AND  CONCLUSIONS 


The  major  purposes  of  the  present  program,  established 
at  the  outset,  were  to  develop  and  demonstrate  a fair  classifica- 
tion/reclassification procedure  for  enlisted  personnel  in  the  Navy 
and  to  demonstrate  the  developed  procedures.  To  these  ends,  two 
recent  concepts  were  applied  to  provide  the  basic  foundation  for 
the  procedures.  The  first  concept  was  the  assessment  center  ap- 
proach. This  concept,  which  has  become  increasingly  accepted 
in  managerial  potential  evaluation,  has  been  only  seldomly  employ- 
ed in  the  technical  performance  potential  evaluative  sphere.  The 
second  concept  was  based  on  a test  development  model  which  con- 
tends that  the  person  who  demonstrates  the  ability  to  learn  a sam- 
ple of  a job  in  a training  and  evaluative  situation  will,  given  appro- 
priate job  training,  be  able  to  learn  the  total  job. 

This  latter  concept  represents  a novel  test  development 
point  of  view  and  represents  a departure  from  orthodox  test  devel- 
opment models.  While  such  a concept  is  not  believed  to  be  a sub- 
stitute for  predictive  evaluation,  the  concept  is  believed  to  repre- 
sent a practical  approach  to  assuring  commonality  between  pre- 
dictor and  criterion.  Certainly,  the  concept  seems  to  possess 
more  common  sense  than  the  approach,  often  employed,  which  at- 
tempts to  demonstrate  that  predictor  scores  correlate  well  with 
some  other  concurrent  measures.  Two  concurrent  measures  can 
be  highly  correlated,  but  both  may  possess  no  long  term  predic- 
tive validity.  In  this  case,  the  concurrent  validity  concept  seems 
meaningless.  We  suggest  that  the  concurrent  criterion  approach  of- 
ten receives  greater  emphasis  and  is  of  less  value  than  most  test 
constructors  seem  willing  to  admit.  Consider  school  grades,  an  of- 
ten used  (misused?)  criterion  employed  to  establish  the  "validity" 
of  a test.  The  correlation  between  the  predictor  and  school  grades 
is  established  by  the  test  constructor.  The  correlation  between 
school  grades  and  job  success  is  known  to  be  low  both  in  general 
and  in  the  Navy  situation.  With  imperfect  predictor-school  grade 
correlation  and  with  imperfect  school  grade-job  performance  corre- 
lation, there  seems  to  be  a degree  of  fuzzy  thinking  relative  to  defend- 
ing such  a paradigm  relative  to  predictor-job  performance  validity 
(where  predictive  validity  is  defined  in  terms  of  predicting  ultimate 
job  performance).  The  present  demonstration  of  the  ability  to  l^arn 
the  job  seems  more  defensible  and  straightforward. 
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Relative  to  the  predictive  validity  issue,  we  do  not  contend 
that  the  assessments  produced  by  the  TCAC  need  not  be  validated. 
While  the  results  of  the  present  study  indicated  considerable  reli- 
ability and  internal  consistency,  they  tell  us  little  about  the  actual 
performance  in  the  selected  career  fields  of  the  men  evaluated.  A 
Fleet  followup  study  is  being  conducted  in  this  regard  and  seems 
especially  important  in  view  of  the  other  positive  aspects  found  for 
the  TCAC  approach. 

TCAC  was  indicated  to  be  viable  with  respect  to  all  areas  in- 
vestigated. Overall,  the  procedures  were  practical  to  implement 
and  were  well  received  by  the  individuals  assessed.  They  perceived 
the  tests  as  both  fair  and  agreeable.  In  particular,  the  job  learning 
aspect  of  many  of  the  tests  was  well  received.  The  performance 
measures  were  adequately  reliable  and  the  policies  followed  by  the 
administrators  were  consistently  applied.  The  policies  of  the  sev- 
eral administrators  were  in  close  agreement  with  one  another.  The 
exercise  scores  have  only  modest  overlap  with  currently  used  Navy 
tests.  As  a prototype,  the  TCAC,  as  implemented,  seemed  quite 
acceptable.  However,  areas  for  improvement  exist.  For  example, 
a level  of  aspiration  exercise  which  is  more  Navy  job  oriented  seems 
desirable.  The  acceptance  of  the  Reliability  exercise  also  seems  to 
have  suffered  because  of  low  perceived  job  applicability. 

It  has  been  contended  by  others  that  the  performance  approach 
is  uneconomical  because  of  its  imposition  of  excessive  participant 
time  requirements.  To  this  we  answer  that  six  hours  were  required 
to  administer  all  TCAC  exercises  to  each  group.  Administration  of 
the  usual  Navy  tests  requires  about  four  hours.  In  view  of  the  po- 
tential power  of  the  TCAC  approach,  the  additional  two  hours  re- 
quired by  the  performance  approach  does  not  seem  excessive.  More- 
over, the  performance  approach  need  not  be  administered  to  all  en- 
T sted  personnel.  Those  who  achieve  an  A school  recommendation 
by  the  usual  classification  procedures  do  not  require  extra  concern. 
However,  those  who  might  in  fact  do  well  in  a career  field  and  do  not 
receive  such  acknowledgment  from  the  usual  procedures  certainly 
deserve  a "second  chance.  " The  TCAC  approach  could  provide  this 
second  chance. 


The  preferred  composition  of  the  evaluative  team  remains 
open.  The  present  program's  administrator  team  was  composed 
of  two  psychologists  and  two  Navy  Chief  Petty  Officers.  It  was  an- 
ticipated that  the  psychologists  would  provide  appropriate  personal- 
ity-social/motivational  appraisal  and  test  administrative  expertise. 
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On  the  other  hand,  the  Chief  Petty  Officers  were  anticipated  to  pro- 
vide job  detail/career  field  expertise  not  possessed  by  psychologists. 
Yet,  the  data  indicated  all  four  administrators  to  be  employing  con- 
sistent policies,  i.  e.  , similarly  relying  on  and  weighting  the  exercise 
scores  to  derive  recommendations.  The  assessor  team  may  have  be- 
come quickly  homogenized  or  the  various  points  of  view  may  not  have 
been  fully  represented  by  the  assessors.  Alternatively,  the  antici- 
pated points  of  view  may  have  been  involved  subtly  but  not  reflected 
by  the  quantitative  procedures. 

There  was  also  some  indication  that  at  least  a part  of  the 
group  decision  making  could  be  replaced  by  a mathematical  tech- 
nique. This  would  further  streamline  the  process. 

The  TCAC  was  designed  with  culture  fairness  in  mind.  To 
this  end,  oral  and  performance  exercises  were  emphasized  and 
hands  on  training  was  implemented.  There  was  minimum  emphasis 
on  reading  skills. 

Finally,  we  do  not  contend  that  the  single  "best"  individual 
or  combination  of  exercises  was  developed.  While  substantial  merit 
was  demonstrated  for  the  exercises  employed,  alternative  or  im- 
provements in  the  present  exercises  are  entirely  possible.  Proto- 
types by  their  very  nature  serve  to  prove  a concept.  Full  produc- 
tion models  often  depart  from  design  specifics  incorporated  within 
a prototype. 

The  data  presented  throughout  the  body  of  this  report  warrant 
the  following  conclusions: 


• The  TCAC  concept,  including  the  job  learning- 
testing and  multiple  assessment  aspects,  repre- 
sents a potentially  useful  and  workable  classifi- 
cation/ reclassification  approach. 

• The  exercises  and  methods  included  in  the  TCAC 
were  such  as  to  allow  acceptable  within  and  be- 
tween administrator  reliability. 

• The  individual  administrators  each  showed  high 
consistency  in  making  judgmental  evaluative  de- 
cisions. 
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• Each  of  the  three  job  analytically  based  dimen- 
sions--learning,  psychophysical-motor,  and 
social/  motivational--considered  in  deriving 
career  field  recommendations  was  relatively 
independent  and  useful  for  achieving  the  re- 
quired classifications. 

• A consistent  policy  was  employed  by  the  TC'AC 
in  forming  career  field  recommendations  and 
in  predicting  job  performance  success. 

• Assuming  adequate  evidence  of  predictive  valid- 
ity, the  TCAC  concept  could  make  a significant 
contribution  towards  increased  effectiveness  of 
personnel  utilization  in  the  Navy. 
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